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ABSTRACT 

This document provides a developer /tester with the knowledge to create and 
run automated tests against a diverse range of ESXi-based test resources. 
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1 DART Automated Test Execution Technology Overview 

1.1 What is Tyrant? 

Tyrant is the name given by Lockheed Martin Advanced Technology Labs to a suite of technology it 
developed for running automated software tests. It is one of the major components of the complete 
DART (Dynamic Automated Range Testing) tool suite. (The other major component handles building out 
cyber test ranges.) The Tyrant suite allows multiple simultaneous developers and testers to run tests in a 
reproducible manner across a wide array of resources. The Tyrant suite also allows administrators to 
manage these test resources. Using Tyrant technologies, one can encode the logic of a test to perform 
against one or more test resources as well as the logic to determine whether the test is a success or 
failure. One can then run this test against a specific set of resources (i.e. the resources needed to run a 
single instance of the test), or have multiple instances of the test run against a diverse range of resources 
to evaluate the functionality of a system-under-test against the whole range of systems it may be run on 
in the real world. Finally, multiple developers can perform these tests simultaneously against a shared 
set of resources without conflicting with each other. 

1.2 Technical Components Overview 

Tyrant is made up of six central technical components: 

• Palantir (Testing Interface) 

• Undermine (Test Script Harness) + Test Scripts 

• Overmind (Test Script Scheduler) + Test Plans 

• Overview (Test Resource/Result GUI) 

• Reaper (Test Resource Sanitizer) 

• Remote Commit (Remote Job Submission) 

Each component is standalone from the others, allowing for each user to determine the combination of 
components most appropriate for their testing scenarios. 

1.2.1 Palantir 

Palantir provides a common, cross-platform test interface to each of the computer resources in the test 
environment that are running commodity operating systems (e.g. Windows, Linux, OSX, FreeBSD, etc.). 
This allows test script writers to write more cross-platform test scripts, expecting a consistent interface 
(e.g. “put file”, “execute”, “spawn”, etc.) on each of the computers needed for the user’s test scenario. 
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1.2.2 Undermine + Test Scripts 

Undermine is a test script harness for executing the user’s test scenarios. The user will encode their test 
procedure in a “test script”. Then he/she will run that test script via Undermine, which will effectively 
automate their procedure. Undermine performs the actions on the test resources via Palantir if it exists 
on the test resource. 


TEST RESOURCES 



1.2.3 Overmind + Test Plans 

Overmind examines the set of test resources in the Test Resource section of the Overmind Database and 
schedules tests to be run across those resources. The user describes what tests they want to run, the 
resource constraints, and the desired iteration and/or replication of each test script in a “test plan”. 
Multiple users can submit test plans concurrently, and Overmind will schedule each possible test to run 
when possible. Overmind runs an instance of Undermine for each test that needs to be performed. 
When tests are completed Overmind puts the test results into the Test Result section of the Overmind 
Database and marks the resource as requiring sanitization in the Test Resource section of the Overmind 
Database. 
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1.2.4 Overview 

Overview is the web-based GUI that allows a user to view currently running and past test results in the 
Test Result section of the Overmind Database previously inserted by Overmind as well as graphically 
manage the resources in the Test Resource section of the Overmind Database. 
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Namespaces (test plans) 


Namespace: (>)|(*) 


Limit: 200 [ok] « Prev Nexl» 

NS Notes: 

E 


FUten n_name~- Utt. 1 

Test Plan: 


w 

Test Case: 



Refresh: 1 START ) 



1.2.5 Reaper 

Reaper monitors the Test Resources section of the Overmind Database for resources that require 
sanitization and performs a custom action to sanitize the resource. The custom action is often 
something as simple as reverting a VM to a previous snapshot, but it can be as custom as rebuilding the 
entire operating system on that resource from scratch. Different sanitization modules can be specified 
for each resource if desired. The same sanitization module is used for all of the resources in the same 
hardware class. After the sanitization process completes, Reaper marks the resource as clean in the Test 
Resource section of the Overmind Database, indicating to Overmind that the resource can be considered 
for use in a test again. 
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1.2.6 Remote Commit (Remote Job Submission) 

While there are many ways to install and configure the Tyrant software, one very common setup is the 
one that allows for multiple users to submit and run their own tests on a set of common testing 
resources. 

In this scenario each user has their own local copy of Overmind, Undermine, his/her tool or System 
Under Test (SUT), and test scripts and test plans relative to that SUT. Each user runs the 
remote_COmmit tool to copy those five components from their local box to the main testing server, 
start Overmind, and submit the necessary test plans. This ensures that each user can be running 
completely different tests from every other user. This also allows each user to continue their 
development of any of those five components without affecting currently running tests. 

Each user can then browse to the Overview web GUI to monitor progress of their running tests. 

The following picture illustrates all of the Tyrant components working together to execute this CONOR 
(“P” == Palantir, “UM” == Undermine, “OM” == Overmind) 
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1.3 For the Developer 

For the developer, Tyrant provides the tools to create and run automated test scripts (in undermine) and 
test plans (in overmind), view test results, and work simultaneously on a range of resources. This 
manual covers how to use a range of ESXi virtual machine resources previously set up by an 
administrator and how to create and run tests. 

1.4 Repository Structure 

The core of Tyrant is provided in two Mercurial repositories: tybase and tyworkflow. The tybase 
repository provides the tools used for running single tests. The tyworkflow repository provides the tools 
used for running tests across ranges of resources and managing these ranges. The tybase repository is 
standalone, but the tyworkflow repository has dependencies on tybase. 

With the split of functionality between tybase and tyworkflow, it can sometimes be difficult to 
determine which repository you should be in to perform certain operations. Unless specifically directed 
otherwise by this manual, the following are a few general rules to help you determine the correct place 
to perform certain operations: 

• If the operation relates to running an individual test against specific test resources without use 
of any range scheduling (e.g. overmind), you should be in tybase. 

• If the operation relates to linking in collections of test scripts (leafbags), you should be in tybase. 

• If the operation relates to scheduling tests to run on a range or managing a range of shared 
resources, you should be in tyworkflow. 

The magnum repository contains code used for testing with PSPs (Personal Security Products) and for 
automatic software updating. It also contains support for detecting PSP events by watching a resource’s 
screen for changes. Related to magnum are the PIL repositories (PIL-linux-i686 and PIL-linux-x86_64). 
These provide the Python Imaging Library, which is used by magnum event detection to find changes 
between screenshots. Two exist because they contain compiled code which needs to match the 
architecture of the system it will run on. 

In-depth descriptions of the contents of each repository are included in Appendix B . 

1.5 Tools 

This manual covers the following Tyrant tools, which will be used and configured by the range 
administrator (with the repository containing each tool in parentheses): 

palantir (tybase): A component which runs on each test resource, allowing the resource to be 
controlled by test scripts via a TCP connection. Palantir serves on ports 51134 and 51135 (for 
Windows) and 51134 (for Linux). 

undermine (tybase): The component which runs a single instance of a test. Undermine connects via 
palantir to the resource(s) used by a test and runs the given test script. 
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overmind (tyworkflow): The component which schedules tests to run simultaneously. Overmind 
uses a database of test resources and schedules these resources for incoming tests. 

reaper (tyworkflow): The component which handles reverting a resource to a clean state prior to 
running a test. What this means depends upon the reaper module in use for a given resource. 
For a virtual machine, this usually means reverting to a snapshot, but for a physical computer, 
this could mean using some imaging or auto-building solution to build out a certain OS 
configuration on a resource on-demand for a test. 

overview (tyworkflow): The component which displays test results and allows some management of 
test resources via a web-based interface. 

process_plan (tyworkflow): A command-line tool used to process test plans (specifications for 
running multiple instances of tests against a specified variety of resources). This lets you submit 
plans to be run or see how many combos (specific runs of a test on specific resources and with 
specific parameters) would be run. 

remote_commit (tyworkflow): A tool which allows remote submission of tests to a central Tyrant 
environment. While overmind on its own allows simultaneous tests to be run against a range of 
resources, remote commit makes overmind useful for a team of developers running different 
sets of code. 


1.6 Directory Structure 

Because each setup will involve testing different components, users will need to create a directory 
structure similar to the following: 



Symlinks can be made between the various repositories using the standard In - s command. 
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Additionally, a user should be in the base of either tyworkflow or tybase to run specific commands. For 

example, if a user was running . /bin/undermine, he would be doing so out of the tybase directory. 

If he was running . /bin/ remote_commit, he would be doing so out of the tyworkflow directory. 

2.7 Assumptions 

This document makes some assumptions about the type of setup desired. Specifically: 

• It is assumed that testing will be performed on VMWare ESXi virtual machines. 

• It is assumed that remote commit-style testing will be performed (with a team of developers, 
each at their own workstation, remotely submitting tests to a central server). 

• It is assumed a mysql database will be used for storing resource and test information rather than 
sqlite3. 

• The remote commit environment on the test server will run as root, and developers will submit 
their tests as root on the test server. 

• The reader is familiar with working on the Linux command line and programming in Python. 
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2 Environment Setup 

Before we can get to the work of writing and running tests, the environment must be set up. We will 
perform the setup now and verify its correctness in later sections. 

On your developer workstation, clone the tybase and tyworkflow repositories into the same base 
directory. In the tyworkflow directory, run make. This links tyworkflow to tybase and unpacks our built- 
in python distribution. 

A few configuration files need to be modified to enable you to use the range that’s been set up for you. 

In tyworkflow, copy the file rc/def aults/db . rc to rc/. In the copied file, make the following 
changes (you may need to talk to the administrator to get this information): 

• set resource_manager/dbname to the name of the overmind database 

• set resource_manager/engine to mysql instead of sqlite3 

• set mysql/host to the hostname or IP address of the test server, which is where the overmind 
database resides 

• set mysql/user and mysql/passwd to the username and password used to access the 
database you set in resource_manager/dbname 

Also, copy rc/ defaults/ remote_commit . rc to rc/ and make the following changes: 

• remote_commit/remote_user: the user to connect to the testing server as (via SSH when 
syncing up files or running commands; defaults to root, which is typical) 

• remote_commit/remote_host: the hostname or IP address of your testing server 

• remote_commit/remote_commit_dir: the path on your testing server to the commits 
directory (/proj /testing/commits is the example used in the administrator manual). 

For the above settings, make sure you uncomment any options which you set which are currently 
commented. That is, remove the semicolon from the beginning of any option line that you modify. 

2.1 Viewing Resources 

A simple task to start with is to view the list of resources. Navigating to the test server's /overview 
directory (e.g. http://testserver.example.com/overview) will present you with a menu of overview pages 
(Namespaces, Recipes, Resources, Management). If you click on Management, you should see a table 
listing all the resources you imported in the previous step. 

2.2 Reserving Resources 

When performing maintenance, one of the first things to do is reserve the computer you'll be working 
on. Reserving a computer prevents it from being scheduled for tests, so that it doesn't suddenly get 
used while you're working on it, and so that any changes you make to it while servicing it don't cause 
problems for developers' tests. 
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To reserve a computer in overview, navigate to the Management page (e.g. 

http://testserver.example.com/overview/add-computer.php). This page lists all the resources in the 
range and allows various tasks to performed on them. Check the box on the row for the computer you 
want to reserve, enter your name or some other sensible identifier in the “Reserver's Name” field on the 
left side of the page, and click the “Reserve” button. When the page refreshes, the resource you 
selected will have a yellow background for its row and the "use” field will say “N”. To un-reserve a 
resource, making it available for testing again, check the resource's box and click the “Use Testing” 
button. The row will lose its yellow background and “use” will say “Y”. 

3 Leafnodes (Test Scripts) 

Test scripts (known as “leafnodes” in Tyrant lingo) are the main units of work performed by undermine. 
Leafnodes typically involve one or more assets with palantir installed on them (but not always). 
Leafnodes can be as simple as a Python function that operates on some input parameters, to as complex 
as a class that choreographs a set of actions on multiple remote hosts via palantir. Where practical, 
leafnodes should be made smaller rather than larger to facilitate reuse. 

3.1 Leaf node Concepts 

Leafnodes come in different types (detailed below), but share some or all of the following concepts, 
which will be helpful to keep in mind for the rest of this manual: 

• hosts: The test hosts, or resources against which the leafnode works. Some leafnodes do not use 
any resources, but most do, as running tests on computers is a primary purpose of leafnodes. 

• inputs: If you think of the leafnode as a function, these are the arguments. 

• progress messages: Asynchronous messages the leafnode can output during the course of its 
execution. 

• result code: An indicator of the status of the leafnode returned when it's finished (e.g. success, 
failure, error). 

• result (or output): A return value from the leafnode (different from its status). 

3.2 Creating and Running a Simple Leafnode 

We’ll start by creating a simple leafnode, then having done that, delve into the details that will let you 
create more powerful test scripts. 

To start, inside yourtybase directory, create the directory path leaf bags/tutorial/tutorial 

(the duplicate tutorial in the path is intentional). In this directory, create an empty init . py 

file to make it a valid python package. 

Open up your text editor of choice and enter the following text: 

from tybase . undermine . leaf import Leaf 
from tybase. undermine. meta import leafi 
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@leaf i . Def ineActuator ( ) 
class Hello(Leaf ) : 
def run(self ) : 

host = self . hosts[0] 

ret = host . execcmd( ' echo ' , self .args[0] ) 
host . fwrite( 1 /tmp/test . txt 1 , self . args[0] ) 
dat = host . fread( ' /tmp/test . txt ' ) 
if dat.strip() == ret.strip(): 

return self . SUCCESS, 'cmd output matched file contents' 
else: 

return self . FAILURE, 'cmd output differs from file contents' 

Save this as leaf bags/tutorial/tutorial/hello_world . py. 

In your web browser, navigate to the overview web interface running on the test server (e.g. 
http://testserver.example.com/overview). This is the interface by which you will later be able to view 
your test results, and will be covered in detail later. For now, click the Management link. Select a Linux 
resource whose “status” column says “avail”, check the box next to it, put your name in the “Reserve 
Name” field on the left-hand side of the page, and click “Reserve”. This reserves the machine for your 
use so that it won’t be scheduled for others’ automated tests, preventing your work from messing up 
others’ or vice versa. Note the IP address of this resource. 

In your tybase clone, run the following command, where IP_ADDR is the IP address of the resource you 
reserved: 

bin/undermine tutorial . hello_world . Hello IP_ADDR -- "hello, world" 

Here's an example of the output you should see: 

2013-09-23_16 : 51 : 02 . 44 (00191) [INF] script 22331: output_dir: 

. /output/undermine/nlsheppa/2013_09_23-16_51_02_348754 

2013-09-23_16 : 51 : 02 . 44 (00191) [INF] script 22331: COMPLETION: 
success 'hello, world' 

If you look on the actual Linux VM you reserved, you'll find a file / tmp/hello . txt with contents 
“hello, world”. 

You have written and run your first leafnode. Unreserve the resource you reserved previously by going 
back to the Management page, checking the box on the resource you reserved and clicking the “Use 
Testing” button. 

3.3 Leaf nodes in Depth 

3.3.1 Writing Leafhodes 

The general idea of writing a leafnode is that you write some sort of callable (see below) which receives 
zero or more palantir client objects, arguments and keyword arguments, performs some operations with 
them, and then returns a result code (see below) and result value, optionally with some progress 
message along the way. The callable is decorated with various decorators to define "metadata" for the 
leafnode. 
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In this section, we first illustrate the different ways of writing leafnodes, then dive in to the details of 
how to define metadata on them. 

3. 3. 1.1 Types of Leafnodes 

Here we cover the two main styles of leafnodes, class-based and function-based. 

3.3. 1.1.1 Type 1: class-based leafnodes 

Class-based leafnodes are the most powerful and consist of a class which inherits from the Leaf class 
provided in tybase (by importing tybase. undermine. leaf.Leaf) and overrides certain methods (all of 
which take no arguments other than the self reference to the instance they're bound to). The methods 
which the leafnode may override are: 

• r unSetup: Run before the body of the leafnode. If this raises an exception, the leafnode will 
stop and the SKIPPED result code will be returned. 

• run: The body of the leafnode. If this raises an exception, the leafnode will stop and the ERROR 
result code will be returned. Otherwise, this method must return a tuple of the result code and 
output value of the leafnode. 

• r unCleanup: Run after the body of the leafnode in all cases except when the leafnode times 
out, regardless of the leafnode's result code. If runSetup has an error other than a timeout, 

r unCleanup is still run (so runCleanup is similar to the finally block of try . . . except 
. . . finally). The success or failure of runCleanup does not affect the final result code of 
the leafnode. If the leafnode body returns SUCCESS, the final result code will be SUCCESS 
even if runCleanup throws an exception. 

• stopHandler: Run in the case of a timeout, when the leafnode is being stopped. This method 
has a limited time (currently five minutes) in which to run, which is why it's separate from 
runCleanup, which doesn't have as small a time limit. 

In this style of leafnode, self is your reference to the currently running script. You access your host(s) 
through self . hosts, which is a list of palantir client objects (even if the leafnode only takes one 
host). Your arguments are provided initially in self . args (positionally-specified arguments) and 
self . kwargs (arguments specified with keywords) without regard to what's defined in the input 
parameters. You'll probably want to normalize your args by calling either self . normalize_args or 
self . normalize_kwargs in your run method (see below). 

Here's an example python script illustrating this type of leafnode (the DefineActuator and other 
metadata decorators will be covered later): 

from tybase . undermine . leaf import Leaf 
from tybase . undermine . meta import leafi 

@leafi. DefineActuator ( ) 
class MyLeafnode(Leaf ) : 
def runSetup(self ) : 

#here you do setup tasks, like perhaps installing some 
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#supporting piece of software on an asset 
pass 

def runCleanup(self ) : 

#here you do cleanup, like perhaps deleting some temporary 

#files 

pass 

def run(self ) : 

#the body of your leafnode 

#normalize to a diet of kwargs so we get our default values 

#and everything easily accessible by name, even if the args 

#w ere given positionally 

self. kwargs = self . normalize_kwargs( ) 

#a common thing to do if you only have one host 
host = self .hosts[0] 

#do some testing stuff 

#perhaps we want to measure how much data was exchanged over 
#the network in this case, our result value is an integer; 
#data type specification will be explained further on 
traffic_size = some_measurement_f unction ( ) 

return (self .SUCCESS, traffic_size) 

3.3. 1.1.2 Type 2: function-based leafnodes 

When you have a simpler testing task, you might choose this second type, in which you simply write a 
python function, which you optionally decorate with some metadata. This type of leafnode is simpler, 
but also less powerful. Differences compared to class-based leafnodes are: 

• Your reference to the currently running script is stored at the context attribute of a host 
object. This unfortunately means that if your function-based leafnode takes no hosts, you will 
not have access to a reference to the currently running script and will not be able to do things 
which require it, like running a sub leafnode. 

• You cannot define setup and cleanup logic like you can with runSetup and runCleanup for class- 
based leafnodes. 

• You have less flexibility in setting the result code of your leafnode, as follows: 

0 If your function raises an exception, ERROR will be returned, with the exception as the 
result value. 

0 If your function returns at all (whether True, False, a number, None, anything), then 
SUCCESS will be used. 

• Input parameters are typically defined implicitly by the function prototype, rather than explicitly 
with metadata on the leafnode. 
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• Arguments are handled differently. The arguments that are passed to your function-based 
leafnode consist of the host objects, followed by the args, followed by the kwargs. Thus, if your 
leafnode is called with too many or two few hosts, you can end up with an argument you 
expected to be a host containing an arg value (too few hosts), or an arg containing a host rather 
than the arg value you expected (too many hosts). 

• Hosts, args and kwargs are accessed by the names you give them in the function prototype, as 
with any function. 

Here's an example python script illustrating a function-based leafnode: 

import tybase. undermine. meta.leafi as leafi 

def my_leafnode(hostl, host2, argl, arg2, kwargl=0, kwarg2=True) : 

#do some testing stuff 

#no matter what you return here, the result will be SUCCESS (as 
#long as you actually return and don't throw an exception) 
return True 

3. 3. 1.2 Leafnode Metadata 

Leafnode metadata is how you define things like what kinds of assets your leafnode works against, what 
kinds of data it takes on input and output, whether it provides asset properties, etc. Metadata is set on a 
leafnode by decorating the leafnode with decorators provided in the tybase.undermine.meta. leafi 
module. Metadata is not strictly required to run a leafnode, but it is advised, and it is necessary to take 
advantage of certain advanced leafnode features. This section's subsections explain the provided 
metadata decorators organized by the purpose they serve, with the name of the decorator in 
parentheses. 

3.3. 1.2.1 Defining the Leafnode Purpose (DefineActuator, DefineSensor, DefineProcessor) 

These three mutually exclusive decorators describe the function the leafnode serves and how it will 
interact (or not) with any assets it uses. Currently the usage of these decorators is only by convention; 
they don't do anything special to the leafnode you put them on (except for DefineSensor with asset 
properties). However, leafnodes need to have some type of metadata, and putting one of these on the 
leafnode is a good way to satisfy that requirement. 

The convention for these decorators is: 

• DefineActuator: leafnodes that make changes to an asset (e.g. delete a file, install a piece of 
software, etc) 

• DefineSensor: leafnodes that only query information on an asset, not make changes to it. If you 
want your leafnode to assert (provide) asset properties, it must have this decorator. 

• DefineProcessor: leafnodes that do not care about what assets they receive, and may not even 
take any hosts at all 

3 . 3 . 1 . 2 . 2 D efining Input Parameters (Inputs ) 

This decorator takes as its arguments tuples defining 
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• the name of an input parameter 

• its data type (see below for valid types and how to specify them) 

• optionally a default value for the parameter. 

This decorator only makes sense for the first type of leafnode (class-based), since the other two types 
depend on the arguments defined in the function prototype. 

An example usage is: 

import tybase. undermine. meta.leafi as leafi 

@leafi. Inputs( 

( 'num_runs' , int), 

('interval', float, 5.0), 

('path', str, 'C:\\test_dir ' ), 

( ' quick_run ' , bool, False) 

) 

class MyLeafnode . . . 

3.3.1.2.2.1 Input and Output Data Types 

Leaf node parameter data types can be any scalar type that can be pickled, as well as lists or structs. 

For scalar types, the data type definition (the second field of the input parameter tuple, or the argument 
to the FinalOutput decorator) is simply the type. For example: 

@leafi . Inputs ( ( ' num_runs ' , int, 5) ) (defines a single input parameter of type int 
named num_runs with default value 5) 

@leafi. FinalOutput (bool) (defines a result value of type bool) 

Typical scalar types are str, int, float and bool. 

For complex types, you show the data types of the scalar parts of the complex types in the context of 
that type (as a list for lists, as a diet for structs). Also, complex types may be nested. For example: 

@leafi . Inputs ( ( ' animals ' , [str] ) ) (defines a single input parameter which will be 
a list of strings named animals and have no default value) 

@leafi.FinalOutput({'MemTotal' : int, 'MemFree': int, 'Swapfile' : str}) 
(defines a result value which will be a struct with three fields of type int, int and str, respectively) 

§leafi . FinalOutput ([{' name ' : str, 'lat': float, 'long': float}] ) (defines a 
result value which will be a list of structs, each representing a city) 

@leafi.FinalOutput({'size' : int, 'files': [str]}) (defines a result value 
which will be a struct of a size value and a list of filenames [perhaps this is the size and contents 
of some archive file the leafnode processed]) 
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However, valid leaf node lists and structs are more limited than what can be expressed in Python lists 
and diets, as follows: 

3.3.1.2.2.1.1 Lists 

Lists are defined by giving the type of the scalar values of the list in list context in the input/output 
definition, as seen in the above table. Thus, every element contained in a list must be of the same type. 

If you need to pass a static set of values of different types, consider using a struct instead. If you really 
need to pass a variable number of items of different types, consider (1) a list of structs, or (2) multiple 
lists, each of which contains a different type. 

3.3.1.2.2.1.2 Structs 

Structs are defined by giving a diet whose keys are the names of the struct fields and whose values are 
the scalar types for each held. The difference between the leafnode struct type and regular Python diets 
is that structs are defined with a static set of fields. If you really need to input (or output) a dict-like data 
structure, here are a couple of options: 

( ' keyValsl ' , [[str]]), # only if key and value types are the same 
('keyVals2', [{'key':str, ’val':int}]) # different types for key & value 

Even with the above alternatives, leafnodes are still restricted in that they cannot input or output 
arbitrary types. 

3. 3. 1.2. 3 Deriving Inputs from Function Introspection (Derivelnputs) 

For function- and method-based leafnodes, where input parameters are based on the function/method 
definition, this decorator will use function introspection to automatically generate input parameter 
metadata (as you would specify with the Inputs decorator for class-based leafnodes) from the 
function/method definition. You simply provide this decorator with no arguments, like so: 

@leafi.Derivelnputs( ) 
def my_leafnode( . . . ) 

3. 3. 1.2.4 Defining the Result Data Type (FinalOutput) 

The data type of the result value, aka output, is defined using the FinalOutput decorator, which takes the 
data type specification as its argument. This decorator uses the same specification as the Input 
decorator, as explained above. 

3.3. 1.2.5 Defining the Progress Message Data Type (ProgressOutput) 

The progress message data type is defined just as with the result data type, but using the 
ProgressOutput decorator instead of the FinalOutput decorator. 

3.3. 1.2.6 Defining Leafnode Alias (Alias inside of a Define*) 

The Define* decorators do have another purpose. Inside of a Define* decorator, you can also specify an 
alias with the Alias class provided in the leafi module. This is used with polymorphic leafnodes. You 
define an alias like so: 
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@leafi . DefineActuator (leafi . Alias( ' uber_leafnode ' ) ) 
def my_leafnode( . . . ) 

3. 3. 1.2. 7 Defining the Leafnode "Subjects" (assets to run against) (Subject) 

Using the Subject decorator, you can define constraints to restrict what your leafnode can run on. These 
constraints utilize asset properties (a list of which can be found here). In the Subject decorator, with the 
constraints keyword, you specify a list of constraint comparisons with the Prop, AndProp and OrProp 
classes provided in the leafi module. All the elements of the constraints list must match for the leafnode 
to be allowed to run. 

Some examples: 

Require 64-bit Windows 7: 

@leafi. Subjects 
constraints=( 

leafi. Prop( 'sw. os. architecture 1 ) == 'x86_64', 
leafi. Prop( 'sw. os. name' ) == *6.1', 
leaf i. Prop ( 'sw. os. family' ) == 'Windows' 

) 

) 

Require 64-bit Windows 7 (illustrating the use of AndProp, which is unnecessary, but valid): 

@leafi.Subject( 

constraints=( 

leafi. AndProp( 

leafi . Prop( ' sw. os . architecture ' ) == 'x86_64', 
leafi. Prop( 'sw. os. name' ) == '6.1', 
leafi. Prop( 'sw. os. family' ) == 'Windows' 

), 

) 

) 

Require 32-bit Windows 7 or XP (illustrating the use of OrProp): 

@leafi.Subject( 

constraints=( 

leafi . OrProp( 

leafi. Prop( 'sw. os. name' ) == '5.1', 
leafi. Prop( 'sw. os. name' ) == '6.1' 

), 

leafi . Prop( ' sw. os . architecture ' ) == 'x86_64', 

) 

) 
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3.3. 1.2.8 Defining the Default Leafnode in a Module (MainLeaf) 

If you so choose, you can mark a leafnode in a module as the default leafnode for that module. Then, 
when you specify the leafnode to run (e.g. on the undermine command line), you need only specify as 
far as the module, and undermine will automatically run the default leafnode you marked. To do this, 
place the MainLeaf decorator on the desired leafnode. 

For example, consider the example leafnode we created and ran in Creating and Running a Simple 
Leafnode . If we added the MainLeaf decorator to the Hello class, then rather than referencing the 
leafnode as tutorial . hello_world . Hello like before, you could reference it as simply 
tutorial . hello_world. 

The MainLeaf decorator would be added to the example leafnode like so: 

@leaf i . Def ineActuator ( ) 

§leaf i . MainLeaf ( ) 
class Hello(Leaf ) : 

3.3. 1.2.9 Inheriting Metadata from Parent Classes (InheritMeta) 

With the class-based style, where class inheritance is involved, this decorator can be used to inherit 
metadata defined on a parent class to the child class. You put this decorator on the class and the class 
would inherit metadata from parent classes. 

3. 3. 1.3 Inside the Leafnode 

At this point, we know how to structure a leafnode, but what goes in the body? 

Technically, pretty much whatever you want within the limits of python. Typically, you interact with 
palantir client objects to put or get hies from assets, run commands on them, do operations influenced 
by the input parameters, and so forth. To see a list of the operations available with palantir, run 
bin/ palantir_admin -h in your tybase clone. Each of the methods documented therein are 
accessible on the host objects. For example, to put a hie, you’d do: 
self . hosts[0] . put(src, dest) 

During the leafnode, you may choose to emit progress messages and at the end you return a result code 
and a result value (depending upon leafnode style). 

3.3. 1.3.1 Emitting Progress Messages 

To emit a progress message, you call the emitProgress method on the Leaf class. If you're in a class- 
based leafnode, that means calling self . emitProgress. If you're in a function- or method-based 
leafnode, you need a reference to the currently running script, which is available on your palantir client 
objects as the context attribute (e.g. host . context . emitProgress). This method is defined as 
follows: 

def emitProgress(self , data, seq=-l, tstamp=None, spath=None, dpath=None) 
where the parameters are: 

• data: The value of the progress message, which must match the type you defined in the 
ProgressOutput decorator. 
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• seq: Sequence number of the progress message (defaults to one greater than the last one, 
starting with zero) 

• t Stamp: Time stamp of the progress message, defaults to the current time. 

• spath: Perhaps this means source path, but it appears to have no meaning and is rarely used, if 
at all. 

• dpath: Override the path to the file where the progress message is stored, defaults to a file in 
the output directory whose name includes the sequence number. This is rarely used. 

Usually, you should only provide data. An example call would look like: 

self . emitProgress({ ' currentSpeedMPH ' : 88 . 8} ) 

3. 3. 1.3. 2 Returning a Result 

When in a class-based leafnode, you return a tuple of the result code and the result value. The result 
codes are constants defined on the Leaf class, as enumerated below. The result value is whatever value 
you want, of the type you defined in your FinalOutput decorator. 

When in a function-based leafnode, as explained above, you simply return the result value and the result 
code is determined for you. 

3.3. 1.3.3 Leafnode Result Codes 

The valid leafnode result codes are defined as attributes on the Leaf class and are as follows: 

• SUCCESS: test completed without error and returned desired results (e.g. your software works) 

• FAILURE: test completed without error and returned incorrect results (e.g. your software doesn't 
work, but your test is written properly) 

• ATTENTION: test completed without error and returned generally desired results, but something 
is fishy and you want a human to follow up 

• SKIPPED: test had an error in setup (either in basic undermine stuff like reaping an asset or in 
your optionally provided setup code) 

• ERROR: test had an error while running the body of the test (e.g. your test has a problem and 
threw an exception) 

Based on the convention of these result codes, your code should generally only explicitly return 
SUCCESS, FAILURE, or ATTENTION. If you want ERROR, you should throw an exception describing the 
error. Your leafnode body should not return SKIPPED since that is for setup problems. 

3. 3. 1.3.4 Normalizing Arguments 

Note: This only applies to class-based leafnodes. 

By default, the code that calls your leafnode's functions does nothing to set up the arguments for you. 
That is, when an instance of your leafnode is created, you get some positional arguments and some 
keyword arguments based on how your leafnode was called. Another effect of this is that default values 
are not handled at all, which means you end up with None for the value of any arguments which are not 
provided in the call to your leafnode. So that you don't have to write your own argument handling code, 
two methods (normalize_args and normalize_kwargs) have been provided. These are 
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methods on the Leaf class, so you can access them simply by calling self . normalize_args or 
self . normalize_kwargs from your class-based leafnode. Both take no arguments. 

normalize_args returns the arguments to your leafnode as a list of positional arguments. 
normalize_kwargs returns the arguments as a diet keyed by input parameter name. Both set 
default values for arguments which are not provided (if you gave default values in your leafnode's 
metadata) or otherwise throw exceptions (if you did not give default values). Also, you may only call one 
of these functions (if you call both, then you may get errors about arguments being provided multiple 
times). The idea is for you to set self . args or self . kwargs to the return of the respective 
normalize function (e.g. self . args = self . normalize_args( )). If your leafnode overrides the 

init method, that would be a good place to put it, otherwise you could just put it near the 

beginning of your run method. 

3.3. 1.3.5 Logging Information 

Logging information may be output from a leafnode using the log attributes present on the script class 
and each host object. These log attributes are instances of the python logging module’s Logger object 
(see http://docs.python.0rg/2/library/logging.html for full details on using python logging). Logging 
entries output using a host object’s log attribute are tagged with the IP address or hostname of that 
host. All these log entries will show up in the script.log file in the leafnode instance’s output directory 
(explained further in the "Running Leafnodes” section). 

Examples: 

• Logging an informational message to the script class’s logger in a class-based leafnode: 
self . log . info("something happened") 

• Logging an informational message to the script class’s logger in a function-based leafnode 
(assuming the leafnode takes at least one host and the first parameter is named "host”): 
host . context . log . inf o( "something happened" ) 

• Logging a warning related to a specific host in a class-based leafnode: 

self . hosts[2] . log . warning("something might be wrong") 

3. 3. 1.3. 6 Performing Palantir Operations as a Normal User (Windows Only) 

Normally, when you perform operations on a test resource with palantir (i.e. using methods on one of 
your host objects), that operation is run on the test resource by a palantir processing running as the 
SYSTEM user. If you have an operation that you need to run as a regular user (e.g. because you need to 
interact with the GUI on modern Windows or need the operation to run with limited user privileges), this 
can be done using a system known as emissary which is built into the tybase repository. 

In the body of your leafnode, add a line like the following (this assumes that “host” is the palantir host 
object on which you would run other test operations such as host.put, host.execcmd, etc): 

emhost = host . createEmissary(domain='DOMAIN ' , 
username='USERNAME' , 

password^ PASSWORD' ) 

Replace DOMAIN, USERNAME, and PASSWORD with the domain name, username, and password of the 
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account you want to run as. If the specified user is not logged in, auto-login registry keys will be set and 
the test resource will be rebooted to cause the desired user to log in. If domain, username, and 
password are all omitted, then operations will be run as whatever user is currently logged in. If only 
domain is omitted, then the domain will be ignored when determining whether the correct user is 
logged in and when specifying via the registry what user to automatically log in. 

Once this line of code has run, emhost will be a reference to an instance of palantir running as the 
specified user. This works exactly the same as any other palantir client object; it has exactly the same 
methods and properties, the only difference is what user the remote palantir server is running as. 

3.3.2 Storing Leafnodes (Modules and Leafbags) 

Leafnodes are stored in python modules in what's known as "leafbags". A leafbag has a very specific 
definition which must be followed: A leafbag is a directory which contains python packages. These 
python packages then contain python modules with leafnodes in them. 

When undermine runs a leafnode, the roots of all the configured leafbags are on the python path. This is 
why, when running a leafnode, you can specify it as a "python import path", like you were trying to 
import it in a python script. Deep down in the guts of undermine, that is what is actually happening. 

3. 3. 2.1 Structuring Leafnode Modules 

Leafnodes are stored in python modules containing one or more leafnodes and having one of several 
markers within the first 200 bytes of the tile. When a module has one of these markers, we call it "leafy". 
A module must be leafy in order for it to be scanned when tybase's bin/prepare is run. This marker 
allows the leafnode scanner to quickly ignore modules that have a very low potential of containing 
leafnodes. 

Valid leafy markers are as follows: 

• the string "THIS_IS_A_LEAF_MODULE" 

• the string "#AUTOGENERATED" at the beginning of a line 

• an import involving tybase.undermine.meta.leafi, e.g. 

from tybase.undermine.meta.leafi import fooor 
import tybase . undermine . meta . leaf i 

• an import involving tybase.undermine. leaf 

• an import involving tybase.undermine. main_script 

3. 3.2. 2 Structuring Leafbags 

Your leafbag should have one or more levels of subdirectories and you should put leafnodes in these 
subdirectories (not in the root of the leafbag). In general, you should try to create your leafbags to either 
be entirely self-contained, or to depend on other leafbags (which you would then link in like normal, 
with no added complexity). This makes things easier in the long run. However, this is not always 
practical. If your leafbag depends on other tiles from elsewhere in your project's repository or just on 
other tiles in general, you will need to be aware of this and take it into account when using remote 
commit. See the section on leafbags with non-leafbag dependencies for how to handle this. 
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Since the directories in your leafbag are being treated as python packages, they must have 

init . py hies like any python package. When you run bin/prepare, a component of tyrant 

will scan the leafbag. The leafbag scanner will only recurse into a subdirectory of the leafbag if at least 
one of the two following conditions is true: 

• the subdirectory's init . py file contains the leafbag marker (the comment #LEAFBAG) 

• the init . py file in an ancestor of the subdirectory up to but NOT including the root of 

the leafbag contains the leafbag marker with the RECURSE flag (the comment # LEAF BAG 
RECURSE). The RECURSE flag tells the scanner to go through all the subdirectories regardless of 
whether they have a leafbag marker. 

So, the simplest thing is to just put an init . py in each top-level subdirectory of your leafbag 

with the comment #LEAFBAG RECURSE. If for some reason you have directories in your leafbag 
devoid of leafnodes (such as a large third-party python module), then you might choose to not use 
recursion globally and only put the leafbag marker in specific subdirectories' init . py files. 

3. 3. 2. 2.1 Leafbag structure example 

Consider an example software project (call it eproj) whose developers want to perform automated 
testing with leafnodes. This project may have a code repository (also named eproj) with a subdirectory 
called tests which is the leafbag for this project. With this in mind, consider the following partial 
directory and file structure for the example leafbag: 

• eproj / (root of the repository) 

0 tests/ (root of the leafbag) 

■ utils/ (supporting python modules, not leafnodes) 

• init . py (has no leafbag marker) 

■ net_tests/ (leafnodes for testing the software in a network) 

• init . py (contains leafbag marker) 

• test_data/ (some kind of data used for the tests and some python 
modules to work with it, but no leafnodes) 

o init . py (exists to make this a valid python package, but 

has no leafbag marker) 

■ Standalone_tests/ (leafnodes for testing the software on a single 
computer) 

• init . py (contains leafbag marker with RECURSE flag) 

• win_xp/ (leafnodes for Windows XP) 

0 init .py (no leafbag marker) 

• win_7/ (leafnodes for Windows 7) 

0 init .py (no leafbag marker) 

• notes/ (contains no init . py at all) 

With this structure, the scanner will 

• skip utils/ (since it has no leafbag marker, and it's a top-level subdirectory so there is no chance 
of having an ancestor with a leafbag marker with the RECURSE flag) 
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• look in net_tests (since it has a leafbag marker in its init . py) 

• skip net_tests/test_data (since it doesn't have a leafbag marker and no ancestor has the 
RECURSE flag) 

• look in standalone_tests and any subdirectories (since the init . py in standalone_tests 

has a leafbag marker with the RECURSE flag), BUT... 

• skip standalone_tests/notes (since it has no init . py at all and is therefore not a valid 

python package) 

3. 3.2.3 Linking-in leafbags 

In order to use leafnodes in a leafbag with tyrant, you must link this leafbag in to your tyrant repo. The 
typical way to do this is to create a symlink in the leafbags directory of tybase that points to the leafbag 
you want to use. An alternative is to actually put your leafbag in the leafbags directory of tybase, but this 
usually doesn't make sense from an organizational standpoint because your typical project using tyrant 
has its own repository with the leafbag as a subdirectory. 

So, following the example leafbag above, assuming your current working directory is the root of tybase 
and that the eproj repo is checked out in the same directory as tyrant-dev, you would run the following 
command to link in the leafbag: 

In -s . . / . . /eproj/test leafbags/eproj 
This results in a symlink called eproj in leafbags that points to the leafbag in the eproj repo. 

3. 3. 2. 3.1 Mitigation of naming conflicts 

Note that leafbags can cause naming conflicts. For example, consider two projects, eproj and fproj. 
Suppose that both projects have leafbags with subdirectories named net_tests. In this case, if both 
leafbags are linked in to tyrant at the same time, a naming conflict will occur. The preferred way to 
mitigate this is to add an extra directory level in the leafbags named for the project. For example, in the 
current situation, the net_tests subdirectories that are conflicting are located at 
eproj /test s/net_tests and f pro j /test s/net_tests. To mitigate this, one could add an 
extra directory level to end up with eproj /tests/eproj /net_tests and 
fproj /tests/f pro j/net_tests, respectively. Then, leafnodes in 

eproj /tests/eproj /net_tests would be referenced with a python import path starting with 
eproj . net_tests, and those in fproj/tests/fproj/net_tests would be referenced 
starting with fproj . net_tests. 

3.3.3 Running Leafnodes 

This section deals with running leafnodes apart from the automated workflow and range management 
features provided by overmind. For range testing, see the sections on Test Plans and Remote Commit. 

3.3.3.1 Single Tests 

Undermine, provided in the tybase repository, is the primary tool used to run leafnodes. Undermine lets 
a user run a single instance of a test script against one or more specific resources. The basic usage of 
undermine is as follows: 

bin/undermine <leaf_spec> <host_spec> [<host_spec> ...] -- 
<args_and_kwargs> 
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The command line components are as follows: 

• leaf_spec: This is the specification of the leafnode to run, and may be given as one of the 
following: 

0 python import path to the leafnode: Since all leafbags are on the python path, you can 
give the "python import path" to your leafnode as if you were in python code trying to 
import it. For example, suppose you have a leafnode called ping in the hie 
leaf bags/my_leaf bag/utils/network_f uncs . py (relative to the root of 
tybase). Then, since leaf bags/my_leaf bag is on the python path, if you were in 
python code and wanted to import your ping leafnode, you would say import 
utils . network_f uncs . ping. Therefore, to run this leafnode, you would use 
utils . network_funcs . ping as the leaf_spec. If you put the MainLeaf 
decorator on your ping leafnode, then you could get away with just 
utils . network_f uncs. 

O filesystem path: An alternative way is to specify the filesystem path to the leafnode. This 
may allow you to run leafnodes even when they're not in leafbags (caveat emptor). To 
do this, you just give the filesystem path to the python file containing the leafnode, 
followed by the name of the leafnode in the file, with an '@' in between. For example: 
leaf bags/my_leafbag/utils/network_f uncs . py@ping. As with the 
python import path, if you put the MainLeaf decorator on the ping leafnode, you can 
leave off the @ping component. 

• host_spec: This is a specification of the host to connect to with palantir. Since palantir 
currently only runs over TCP/IP, this must be either the IP address or hostname of the host. 

• args_and_kwargs: Here you specify, space-delimited, the arguments and keyword 
arguments to the leafnode. See the output of bin/undermine -h for an extensive 
description of exactly how arguments and keyword arguments are specified. 

When undermine runs it logs into two files: script.log and undermine.log. script.log contains logging and 
output specifically from the test script, undermine.log contains lower-level logging from undermine and 
palantir, logging which the test script developer may not care about. These files are located in the script's 
output directory. 

When running test scripts standalone with undermine, by default, output directories are stored under 
OUtput/undermine/<USERNAME> relative to the tybase root, where <USERNAME> is the user 
name of the user running undermine. Under this directory, a subdirectory named with the current date 
and time is created, and this timestamp directory is the output directory of the leafnode. Also, in the 
<USERNAME> directory will be a symlink called “latest” which always points to the most-recently-run 
leafnode. This is especially helpful when there are lots of output directories sitting around in a 
<USERNAME> directory. 

3.3.3.2 Batch Testing 

The plundermine tool provided in tybase allows running simple combinations of tests against specific 
test resources. To use it, you give plundermine a leafnode to run, and lists of hosts and parameters for 
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each host and parameter slot the leafnode takes. Plundermine will generate all the possible 
combinations and run them with a level of parallelism (up to a configurable maximum number of 
concurrent undermine runs). In the fairly common degenerate case of a leafnode which accepts only 
one host and no arguments, plundermine is an effective tool for running a given leafnode against a 
whole set of resources. This is useful for some range management tasks. 

See the output of bin/plundermine -h for full details. Some examples are: 

• Run a leafnode which only accepts one host against the three listed hosts: 

bin/plundermine underlib . test_leaf node 

192 . 168 .56.1, 192 . 168 .56.2, 192 . 168 .56.3 

• Run a leafnode which accepts two hosts and no arguments against all possible combinations of 
hosts listed in the two specified tiles: 

bin/plundermine underlib . client_server_test file: clients 
file : servers 

• Run a leafnode which accepts two hosts and two arguments against all possible combinations of 
the hosts from the first tile, the hosts from the comma-separated list for the second host slot, 
and the specified values for the two argument slots: 

bin/plundermine underlib . complex_test file : first_hosts 
192.168.56.1,192.168.56.2 -- one, two, three x, y, z 

For plundermine, output directories are stored in output/plundermine/<USERNAME- 
TIMESTAMP>, where <USERNAME> is the name of the user who ran plundermine, and 
<TIMESTAMP> is the date and time at which plundermine was run. Inside each of these directories 
are numbered subdirectories representing each of the test instances run. These numbered 
subdirectories are each undermine output directories containing the undermine log and data tiles. 

4 Test Plans 

Test plans are the units of work performed by Overmind. They specify what test to run (the leafnode the 
user already has) and what to run it on ("all versions of Windows", "all languages of Windows XP SP2", 
etc). Overmind utilizes the reaper to revert assets to previously stored state (typically, reverting to a 
snapshot on a VM). Overmind stores the results of undermine runs in a database which can then be 
viewed through the overview web gui. Like leafnodes, test plans are written in python and stored in 
leafbags and are referenced on the command line in the same manner (either as python import paths or 
filesystem paths). 

4.1.1 Test Plan Concepts 

The following terms will be useful to know when writing and running test plans: 

• test plan: A python script which defines test cases. This is the thing you run with overmind. 

• test case: A description of what leafnode to run, on what assets, with what parameters. 

• combo or test instance: A specific combination of leafnode, assets and parameters. A test case 
expands into multiple test instances at run time. These test instances represent individual runs 


29 


UNCLASSIFIED 


UNCLASSIFIED 


of undermine. NOTE: In overview, the overmind web gui, test instances are referred to as test 
cases. 

• namespace: An overall identifier in overmind under which multiple plans can run. 

• purge: To immediately cancel a running namespace, plan, or test instance. 

• reap: To revert an asset to a previously stored state. 

• recipe: A definition of an OS which can be placed on a computer (e.g. the family, service pack, 
architecture, language, installed apps) 

• computer: A computer on which a recipe can be installed (e.g. a VM or physical machine) 

• resource: A specific computer with a specific recipe on it. 

4.1.2 Example Test Plan 
In your tyworkflow repository, look at 

src/leaf bag/overlib/preflight/service_ping_plan . py. This is a test plan which runs 
the service_ping_test on all unique combinations of computer and recipe in the range, which for 
our purposes is equivalent to all the resources on the range. We’ll walk through this line-by-line: 

from tyworkflow. support . planlang import * 

This line imports the plan language objects used in writing the test plan 

test = TESTCASE ( 

We begin the definition of a test case which will be used to generate all the actual tests run. 

script = ' overlib . preflight . service_ping_test ' , 

This defines what leafnode to run. You must use the “python import path” method of specifying the 
leafnode to run; do not use a filesystem path. 

hostslots = [HOST ( ) % FACTORS(computer_id=l, recipe_id=l) ] , 
This defines how to generate the actual tests run, or “combos”. This hostslots setting indicates the test 
only takes one host (since a list of only one element is provided), puts no constraints on the chosen host 
(because of the empty argument list to HOST), and indicates that the combination of computerjd and 
recipejd for each chosen host must be unique. This is covered in-depth later. 

samples = -l, 

This line specifies how many of the generated combos to actually choose. The special value -1 indicates 
that all combos should be used. 

namespace = ' preflight-ping-$t 1 , 

This specifies the name of the namespace to use if no namespace is . 

The values defined in the TESTCASE declaration may be overridden on the command line. 

EXECUTE ( 

testcase = test, 

) 

This block sets the testcase defined above to be execute. 
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4.1.3 Parsing Test Plans 

Prior to actually running a test plan, there are a couple operations which may be performed on it to 
verify that it will do what you expect. Because you configured your database by editing rc/db . rc in 
tyworkflow earlier, you can run these steps locally even though your test plans will be run remotely. 

The process_plan command provided in tyworkflow provides the parse and solve subcommands. 
Parse will parse your test plan and give back to you overmind’s understanding of what you’re written. 

You can use it as a quick verification that you wrote what you intended. To parse the example plan from 
above, you would run 

bin/process_plan parse overlib . preflight . service_ping_plan 

If you wanted to parse the plan but then override the samples setting, you could run 

bin/process_plan parse overlib. preflight . service_ping_plan samples=10 
and you would see that change reflected in the output. 

The solve subcommand will parse your plan and then show you what combos your plan would generate. 
To solve the example plan, you’d run: 

bin/process_plan solve overlib . preflight . service_ping_plan 
You could override samples like above, and depending upon how diverse your range is, you may see the 
number of combos decrease. 

4.1.4 Running Test Plans 

The remote commit command (bin/ remote_commit) in tyworkflow allows you to submit your tests 
to a remote test server. Once your test is submitted, you can continue your development in your local 
environment without affecting the test you just submitted. You can even submit tests in parallel, 
allowing you to try one approach to solving a problem, submit a test of it, then try a different approach 
and submit a test for that approach in a different namespace. Here we explain how to run test plans, but 
remote commit has more functionality which is covered in detail later on. 

Test plans are run though remote commit with either the run or r unlite subcommands. These 
commands sync your local environment up to the test server and submit your test to your remote 
overmind instance. These commands take the same arguments as process_plan’s solve and 
parse. To submit the service_ping_plan with remote_commit, you’d run: 

bin/remote_commit run overlib . preflight . service_ping 

If you wanted to limit the number of samples, you would run 

bin/remote_commit run overlib . preflight . service_ping samples=10 

Another useful option is to specify some notes on the namespace with the n_notes argument: 

bin/remote_commit run overlib . preflight . service_ping samples=10 
n_notes="please work" 

4.1.5 Working with a Range 

As a developer, overview will be your primary interface to range testing. Overview lets you browse test 
results and reserve machines for use outside of normal automated testing. 
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4.1.5.1 Seeing Test Results 

To see test results, navigate to overview’s Namespaces page (e.g. 

http://testserver.example.com/test_namespaces.php). Here you’ll see a listing of all the namespaces 
that have been run. When remote commit is in use, there will usually be only one plan in a namespace. 

At each of the list levels, on the right side of the table, you’ll see a summary of how many testcases in 
that entity are in the various states a test can be in (either pending, running, or one of the completion 
statuses). 

The list pages also have forms at the top allowing you to filter by matching on various attributes of 
namespaces, plans, or test cases (specified by naming the field of the table to filter on and the pattern to 
match, e.g. status=error to see all error testcases or n_name~=pref light to see all 
namespaces whose name matches the pattern “preflight”). 

The Refresh field specifies the interval in which any particular page will refresh. This is particularly 
useful for monitoring test results as they finish. 

Finally, the Limit field allows you to specify how many rows you want to be displayed (whether 
namespaces, test plans, or test cases). This prevents loading an entire page with 1000’s of rows if the 
loading time would take too long. 


Namespaces (test plans) 


Namespace: 



Lin*: 200 foie] « Prey | | Next » 1 

NS Notes: 

o 


Test Plan: 


Filter: n_name-= ' fatl 

Test Case: 

(*) c: 

Refresh: | | secs 1 START l) 



- 



start time 

finished 

ir 


1“ 

hIhhuhiih 

[*| 

408 

JacalDettaTest 

34 Finished 

2013-09-17 12:5420 EDT 

34/34 

34 



c 

• 

407 

DdtaTest 

17 Finished 

26 Finished 

2013-09-17 1237:50 EDT 
|2013-09-16 0823:49 EDT 

17/ 17 

26/26 

1“ 

26 



HI 

405 

QsInfoTest 

60 Finished||2013-09-15 16:10:24 EDT 

60/60 

60 




Clicking on a namespace brings you a list of all of the test plans in that namespace. 


Test Plans (test cases) 



Clicking on a plan name brings you to a list of all the test cases in the plan. 
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Test Case Details 


Namespace: 

OsIrfoTest ||405| 

• 

1 results 

NS Notes: 


Test Plan: 

verify osinfo plan-2013 09 17-20 53 40 || 

421 

Test Case: 

verify_osinfo_plan-2013_09_17-20_53_49_955046-421 

616 


Result Code: m 

Start: 2013-09-17 16:53:50 EDT 


End: 2013-09-17 17:07:03 EDT 


Script: underHb.preffight verify_osinfo_test 
Host_0: 1 72.2.2. 122/windows:2003 R2;0;zh-T\V;X86,palantir;vm 


Result: patch_level:spl!=spO 

Output Dir: proj tjTant-de\- output OsInfoTest verify_osinfo_plan-2013_09_17-20_53_40'verify_osinfo_plan-2013_09_17-20_53_49_955046-421 
Undermine Logs run task log trawl 1756 

script-log trawl 981 


udala-call-00000001.dat (ts * 1 144 metajraw) 314 

udata-result-00000001.dat fraw} 20 meta fraw] 311 

udata-result _ code-0000000 1 .dat {raw) 4meta(raw}326 
ndata-timps-00000001 riat trawl 87 meta trawl 315 

undermine log trawl 14976 


HostO 

win2k3r2std-sp0-x86-tv 

172.2.2.122 


The file script .log contains logging output specifically written by the test script developer (by calling 
self . log or host . log in a test script) which undermine . log gives lower-level undermine 
framework logging information. See these files in the case of errors or unexpected results with your 
tests scripts. 

Clicking on any of the log hies leads to the raw data from the file system. 
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/proj/tyrant-dev/output/OsInfoTest/verify_osiiifo_plan-2013_09_17-20_53_40/verify_osinfo_plan-2013_09_17- 

20_53_49_952576-421/script.log 

RegExp Filter: I B 



4.1.6 Test Plans in Depth 

Within the previously outlined basic structure of plan files, there are many constructs in the "planlang” 
(provided in your plans by the line 

from tywor kf low. support . planlang import * 
present at the top of each plan file). Together, these constructs are used to build the specification of 
what tests to run, with what arguments and what types of test resources. 

4.1. 6.1 FILTER 

This is a base class which supports abstract filtering based on values of named attributes. It is extended 
by some other classes in the planlang and is generally not used directly. The typical usage is the HOST 
subclass. 

4.1. 6.2 HOST 

A FILTER subclass that filters resources based on attribute value constraints. The value can be a 
singleton or a list of values. The constraints are specified in the constructor with the general form: 

HOST(<attr>=<value> | [<value>, <value>, . . . ] , <attr>=. . . ) 

For example, 

HOST (family^ 1 windows' , os=['2k', ' xp 1 ] , ossp='sp0') 

HOST objects are the primary means to define the desired set of resources to use for a given test script. 
HOST objects are used in the host slots argument to the TESTCASE constructor, explained later. 

IMPORTANT: The values for "family” and “os” and all of the other fields on which a user can write a 
filter are set by the Overmind database. These values can be viewed using Overview’s "Recipes” and 
“Resources” pages to determine what valid values are. For example, if the administrator sets the "os” of 
a box to be the string “xp_pro” instead of “xppro” and the user wants to run on XP boxes, the filter for 
“os” needs to be "xp_pro” - the exact string match. This loosely-defined schema is nice for rapidly 
adapting to new recipes; however, it does require coordination between the users and the 
administrators. 

We recommend the schema style specified in the recipes . CSV file in the docs/ directory of 
tyworkflow; however, it is more important that an organization is simply be consistent with whatever 
schema they choose. 
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4.1. 6.3 FACTORS 

The class used to control the sampling of test instances. For example, consider a plan with 3 host slots 
and a resource pool of 100 machines. In the worst case, this could generate 100 A 3 potential test 
instances. If each test takes 10 minutes, even with max parallelism of 33 simultaneous test instances, it 
would take a minimum of 10,000,000/33 minutes or ~21 days. In this case, you may want to sample the 
set. FACTORS objects specify the attributes of interest to vary across test instances. Other attributes will 
be randomly selected based on resource availability. The constructor defines the attributes of interest 
with the general form: 

FACT0R(<attr>=True | False, ...) 

For example, 

FACT0R(family=True, os=True, ossp=True, lang=True) 

4.1. 6.4 TESTCASE 

The class used to compose HOST, FACTOR, and parameter values to form a specification for a set of test 
instances. Specifically, a TESTCASE constructor takes: 

• script: Name of test script (leafnode). 

• hostslots: List of FILTER objects, defining the number of host resources and their 
constraints. For example: 

[H0ST(), HOST ( ) ] 

The number of elements in the hostslots list defines the number of resources each testcase 
will use and should match the number of resources the test script defined in the script argument 
requires. 

• paramslots: List of parameter values (defined as a list). For example: 

[['a', 'b'], [ 'C ]] 

Combos are generated for each potential value of a parameter slot. In the given example, the 
first parameter slot can be either ‘a’ or ‘b’, but the second parameter slot will always be ‘c’. So, 
for a very simple example range with only one test resource, a plan with this paramslots 
setting would generate two combos, one with arguments ‘a’ and ‘c’, the other with arguments ‘b’ 
and ‘c’. 

• filter: Singleton FILTER object that defines a global constraint over resources. For example: 

HOST ( pool= 1 pname ' ) 

• xattrs: Singleton XATTRS object, defining attribute constraints across host objects. For 
example: 

XATTRS ( vlan= ' same ' ) 

This line would ensure that the hosts are on the same subnet. 

• factors: Singleton FACTORS object, defining sampling attributes. For example: 

FACTORS (0S=1) 

• samples: Maximum number of sample test instances to run. 

• replications: Number of times to run each sample. 

• priority: Numeric priority of test instances (used to sort scheduling queue). 

• namespace: Namespace name to use when storing test results in database. 

• post_ops: List of functions to run as post operations. 
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• n_notes: Informational notes to store with the namespace this testcase will run in. Remember, 
this MUST be quoted if it contains spaces. 

• p_notes: Informational notes to store with the test plan this testcase will run in. Remember, 
this MUST be quoted if it contains spaces. 

4.1. 6.5 EXECUTE 

The class used to define which TESTCASEs to run for the plan file. The separation of TESTCASE and 
EXECUTE allows you to compose TESTCASE separately from specifying which ones to run. The 
EXECUTE constructor takes the TESTCASE object and an optional set of keyword arguments. If the 
optional TESTCASE keyword arguments are provided, they are used to override the value for the given 
TESTCASE. Note this is purely for convenience and EXECUTE( testcase, **plan_ops) is 
equivalent to: 

EXECUTE( testcase / TESTCASE( **plan_ops) ) 

4.1. 6.6 PARSE 

Returns the list of EXECUTE'd TESTCASEs for a given plan file. Any list operator can be applied to this 
list for composing plans from plans. As with the EXECUTE statement optional arguments can be 
provided to override the value for the resulting TESTCASEs. Note this is purely for convenience and 
PARSE(plan_file, **plan_ops) is equivalent to: 

map (lambda x: x/TESTCASE( **plan_ops) , PARSE ( plan_file ) ) 

4.1. 6.7 Planlang Operators 

The language provides a set of operators over objects for plan reuse and simplification. For FILTER 
operators it is best to think of a FILTER object as representing a set, specifically, the set of resources 
that satisfy the constraints. The valid operators are: 


• FILTER & 

• FILTER | 

• FILTER - 

• FILTER % 

• FILTER % 

• TESTCASE 


FILTER: A new FILTER object representing the set intersection of the two. 
FILTER: A new FILTER object representing the set union of the two. 
FILTER: A new FILTER object representing the set subtraction of the two. 
FACTORS: Binds the FACTORS to the FILTER object 
XATTRS: Binds the XATTRS to the FILTER object 

/ TESTCASE: Copy of left TESTCASE with attributes overridden with the 


defined attributes of the right TESTCASE (undefined attributes use the value from the left 
TESTCASE). 

• TESTCASE & FILTER: Copy of TESTCASE with global filter constraint intersected with 
FILTER. 

• TESTCASE | FILTER: Copy of TESTCASE with global filter constraint unioned with 
FILTER. 

• TESTCASE - FILTER: Copy of TESTCASE with global filter subtracted with FILTER. 

• FACTORS + FACTORS: A FACTORS instance with the union of attributes from both 


operands. 
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• XATTRS + XATTRS: An XATTRS instance with the union of attributes from both operands, 
with attributes in the second operand overriding those in the first when the same attribute is 
present in both operands. 

4.1.7 Plans of Plans 

One incredibly useful feature is to generate plans of plans. A typical use case is to have a high level plan 
that kicks off a lot more lower level plans. For example, a regression_test plan could include kicking off 
all of the relevant plans for a single situation. 

To do this, one needs to generate a plan file that calls run_plan on other plans. The arguments to 
run_plan are the following: 

namespace: typically globals() 

search_path: typically the leaf bag containing the plans 

plan_narne: specific plan name to be called by this plan (without “.py” extension) 

maxCount: how many times to run this plan (equivalent to setting “samples=” from the 

remote_commit command line 

For example, 

from tyworkflow. support . planlakng import run_plan 

run_plan(globals( ), "my . leaf bag", "MyPlan", 5) 
run_plan(globals( ), "my . leaf bag", "MyPlan2", 2) 

NOTE: Any command line arguments (e.g. “samples=”) passed to remote_commit will override any 
arguments passed to the test plans themselves via run_plan. 


4.1.8 Remote Commit in Depth 

Remote commit is provided as an alternative method of setting up an overmind test range and 
submitting plans to it which is useful for environments of multiple developers working against a single 
overmind-controlled test range. 

For background, the standard (non-remote-commit) way of doing things is to, on a single machine, run 
the overmind, reaper, and overview servers. The user may choose to set up a MySQL database for 
overmind to store information about assets and test results in, or they may use the default SQLite 
database. The user then links in their testing code as a leafbag in their tybase repository and submits 
plans from their tyworkflow repository (which has tybase linked in at media/tybase). This lends itself 
well to a single developer working on the range, but doesn't work so well for multiple developers on 
different workstations. Under this model, they would have to SSH in to the machine running overmind, 
cd to the appropriate tyrant directory, put their testing code in place, and then submit a plan. Parallel 
work by multiple developers is not favored in this model. 
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In contrast, remote commit gets around these problems. With remote commit, a single MySQL database 
serves as the point of concurrency. Global instances of reaper and overview are run on a server and a 
directory is created on this server to serve as the root for remote committing to. Users set up their own 
local working directories of tybase, tyworkflow, and their SUT(s). When the user submits a test plan with 
remote commit, remote commit rsyncs all their testing code up to the server (in a subdirectory for the 
developer) and runs the overmind in the tyworkflow the user synced up to schedule the user's tests 
(referring to the database to see what's available). The results of the user's tests are recorded to the 
global MySQL database, which all the users can view on the global overview instance. Any files that 
resulted from the test are also stored on disk and accessible from the overview interface. Once the user 
has run remote commit, which only takes as long as is needed to sync their code up to the server, they 
user may continue development in their working directory without affecting the tests that are running 
on the server. By specifying alternate usernames when submitting test plans, the user may even submit 
one test plan while another is already running. 

4.1. 8.1 SUT Preparation 

In order to use a given SUT with remote commit, you must prepare a shell script that provides some 
functions and constants which remote commit will use. These should be placed in a file named 
rcoverrides.sh in the root of your SUT's leafbag. It is important that the functions respect posix 
conventions for return code, i.e. return 0 on success, nonzero on error. All override functions run relative 
to the tyworkflow root. Likely, you'll want some of your functions to run relative to the tybase root. In 
that case, you'll need to do something like pushd media/tybase at the top of your function and 
then popd at the end. 

Functions: 


• _rcoverride_build: Performs any steps necessary to build the SUT or prepare it for testing 
that are not encompassed in the actual testing code. For example, if your SUT is a single C file 
that requires compilation before testing, your _rcover ride_build function might just run 
gcc. If your project is more complicated and has a makefile, your _rcoverride_build could 
run make. 

• _rcover ride_clobber: Called immediately after running a make clobber in 
tyworkflow’s root, as part of the clobber command. Performs a thorough cleanup of the SUT 
directory. Continuing on the example from the previous function, the 

_rcover ride_clobber for a simple one-file C SUT might just delete the binary resulting 
from the compilation encoded in _rcoverride_build, while the implementation for a 
project with a makefile might run make clean. 

• _rcoverride_Stop: Called when running the stop command, immediately after shutting 
down the overmind service. 

• _r cover ride_summary: Called when running the summary command, just after running 
bin / scan_output (which summarizes the undermine runs recorded in the output directory. 
This function allows you to add any custom output to the summary output. 

• _rcover ride_submit: Called during running of the submit command on the testing server, 
immediately prior to actually submitting the desired plan to the running overmind. This allows 
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you to do things like make settings changes to the running overmind conditionally based on 
which test is being run (a specific example would be to increase the maximum number of 
children for certain larger test plans). 

Constants: 

• UNDERMINES: Overrides the default maximum number of children your remote overmind 
process will allow. 

• CLIENT_TIMEOUT: Overrides the default client timeout (maximum running time for various 
interactions with overmind). 

4. 1.8. 1.1 Leafbags with non-leafbag dependencies 

In order for tests to work with remote commit, the tests and all their dependencies must be linked in to 
your environment so that they will all end up on the testing server during remote commit's rsync. For the 
case of a self-contained leafbag, this works trivially: your SUT's leafbag is linked in to your tybase 
repository (in leafbags/), so when remote commit syncs, the leafbag is synced up. When your 
leafbag has non-leafbag dependencies, those dependencies must also be linked in to your copy of tyrant 
so that they will also be synced up during remote commit, and your testing code must be written so that 
it can reference the dependency relative to the tybase root (so that there are no hardcoded paths that 
will break when the entire directory is synced to some other system). Link in your extra dependency 
with a symlink in tybase’s media directory, then write your testing code to refer to the supporting 
components in that location. Then, using the search_media_path function provided by in 
tybase . support . util, you can retrieve the path to the directory you linked in and then do 
whatever you need to with it (e.g. open a file relative to it, add it to the python path so you can import 
from it, etc). Call search_media_path with the name of the symlink you created inside of media, 
and it will return a usable path to your linked-in media. 

4.1. 8.1. 1.1 Example 

Suppose you have a project with a repository called eproj which contains the following two 
subdirectories (among others): leafbag (the leafbag with leafnodes for your project) and data (a 
directory with some sort of supporting files that are used both for your tests scripts in leafbag and by 
other parts of the software). You would link the leafbag in to tyrant's leafbags/ as always. Then, you 
would also put a symlink in tyrant's media/ pointing to eproj's data subdirectory. For example, 
assuming your current working directory is the root of your tyrant copy and eproj is a sibling of your 
tyrant copy, you could run 

In -s . . / . . /eproj/data media/eproj -data 
Then, a hypothetical eproj test script might contain code like the following to make use of files in that 
directory: 

from tybase . support . util import search_media_path 
epdata = search_media_path( ' eproj -data' ) 

with open(os . path . join(epdata, 'seed-001.txt'), ' r b ' ) as seedfh: 
seed_dat = seedfh . read ( ) 
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4.1. 8.1. 2 Shared directories 

It may be that your testing involves some large set of files which don't change very much and can be 
shared among developers. While each developer does need their own copy of these hies on their local 
workstation for any local tests they might be doing, you would rather not have multiple copies of this 
large set of tiles on the testing server (since each developer has their own subdirectory of the commits 
directory to which they would have to sync their own separate copy of the shared files) and would rather 
not have to go through syncing those tiles every time a developer runs a test. 

Remote commit provides a way to handle this. In the remote_COmmit . rc tile, create a section called 
shared_dir s. Each option in this section defines a single shared directory. The name of the option is 
the path to the shared directory relative to the root of tybase (i.e. where the shared directory currently 
resides in the tybase clone in your local testing environment). The value of the option is the path to the 
actual shared directory on the testing server. When you do a remote commit operation that involves a 
sync (run, runlite, sync, or synclite), the paths given as the option names in the 
shared_dirs section are excluded from the sync. After the rsync part of the sync process is complete, 
remote commit will create symlinks inside the remote tybase root (at the paths given as the option 
names) pointing to the paths given as the values to those option names. This saves having multiple 
copies of the large shared tiles on the testing server, however it is your responsiblity to make sure the 
copy of the shared directory on the testing server is kept up to date, since that will not happen 
automatically when a developer does a "sync”. 

4.1. 8.1. 2.1 Example 

Suppose your eproj test scripts require a set of installers and data files that exist in a directory called 
epro j_data. Currently, you make this set of files accessible by putting a symlink named 
eproj_data in tybase's media directory which points to the actual eproj_data directory (so the 
location of that symlink relative to the root of tybase is media/eproj_data) and then writing your 
test scripts to access the tiles out of media. To set this up as a shared directory for remote commit, you 
would do the following: 

• Place a copy of the shared directory somewhere on the testing server. For our example, we'll put 
it at /proj/eproj_data. 

• Add an option to the shared_dirs section of ty workflow’s rc/remote_commit . rc 
whose name is the location of the shared directory inside tybase and whose value is the location 
of the shared directory on the testing server. For our example, this is: 

media/eproj_data = /proj/eproj_data 

• Now, when you do a sync, media/eproj_data will be excluded from the initial rsync 
operation. Then, after that operation completes, a symlink will be created inside the remote 
tybase at media/eproj_data pointing to /proj /eproj_data. 

4.1. 8.2 Usage 

Remote commit provides several commands via the bin/ remote_commit script (in tyworkflow) 
which are used to sync your SUT up to the testing server, run tests, and administer your remote instance 
of overmind. These commands are explained below grouped by use case 
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4. 1.8.2. 1 (Dry)Running Tests (run, runlite, submit, parse, solve) 

The run and runlite commands are the backbone of running remote commit; you can ignore all the 
others and still work effectively with these two commands. They are basically wrappers which run 
several commands under the hood. They both sync your testing environment up to the testing server, 
start your remote instance of overmind, and then submit the given plan. The difference between the lite 
and full versions is that the full version does a clobber and build on the local side before syncing up to 
the testing server, whereas the lite version does not do this. 

Usage: 

bin/remote_commit [-u <USER>] run <PLAN> [<process_plan_opts>] 
bin/remote_commit [-u <USER>] runlite <PLAN> 

[<process_plan_opts>] 

where <PLAN> is the designator for a plan tile (either a python import path or filesystem path). The - u 
option specifies the subdirectory of the commits directory to work out of, and defaults to the current 
username (of the person running bin/remote_COmmit). In the case of run, the developer's testing 
environment will be synced up to the named subdirectory of the commits directory, the overmind in that 
directory will be used, etc. 

Also, since run and runlite end up calling bin/process_plan on the remote side, you may 
override certain testcase attributes just as you would if you were calling bin/process_plan directly. 
This ability to override testcase attributes is available for any of the remote_commit commands that call 
bin/process_plan on the remote side. The one we find most useful is to override the samples 
value to run a subset of a potentially large set of combos. For example, your testplan may generate 100 
combos, but you only want to run a random ten of them. Then, you would do: 
bin/remote_commit run PLAN samples=10 

The submit command simply syncs up your testing environment and submits the given plan without 
running the build step. This is useful if you make local changes ONLY to your SUT’s testing code (or other 
minor local changes which don’t require rebuilding your SUT), because in that case the changes you're 
syncing up won't have any effect on your remote overmind instance, so a restart is not necessary. If, 
however, you’ve made SUT changes that require a rebuild, then submit may not be safe to run; you 
should use run and runlite instead. If you’ve made changes to actual tyrant code, then you need to 
do the sync command (explained later) first to force your remote overmind instance to restart. 

Usage: 

bin/remote_commit [-u <USER>] submit <PLAN> [<process_plan_opts>] 

The solve command allows you to see how many combos your test plan will run when submitted. This 
is analogous to the bin/process_plan solve command used with the classical overmind setup. 

Usage: 

bin/remote_commit [-u <USER>] solve <PLAN> [<process_plan_opts>] 
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4. 1.8.2. 2 Syncing your testing environment (sync, synclite, diff) 

These two commands sync your testing environment up to the testing server. As with run and runlite, 
the full version does a local clobber and build, the life version does not. An added difference between 
sync and synclite is that sync forces your remote overmind instance to restart, whereas 
synclite does not. If you make changes to core tyrant code and want that to take effect on the 
remote side, you need to use sync, since if the remote overmind doesn’t restart, your changes may not 
take effect, depending upon what you changed. Generally, developers and testers won’t be making 
changes to overmind, but if you receive an updated delivery of tyrant code or the administrator makes 
some changes, you may need to run a full sync. 

Usage: 

bin/remote_commit [-u <USER>] sync 
bin/remote_commit [-u <USER>] synclite 

The diff command is provided as a dry run of syncing. It just uses rsync's dry run capability to show 
you what files will be uploaded/changed/deleted when syncing to the testing server. 

Usage: 

bin/remote_commit [-u <USER>] diff 

4. 1.8.2. 3 Administering your remote overmind instance (start, stop, restart, set_children, 
get_children) 

To start, stop, or restart your remote overmind instance, the respective commands are provided. Users 
do not typically use these since they are handled automatically when running run or sync commands. 
During the stop command, the custom _rcover ride_stop function is run, if provided. 

Usage: 

bin/remote_commit [-u <USER>] start 
bin/remote_commit [-u <USER>] stop 
bin/remote_commit [-u <USER>] restart 

The set_children command allows you to set the maximum number of undermine processes your 
remote overmind process will run in parallel. 

Usage: 

bin/remote_commit [-u <USER>] set_children NUM_CHILDREN 
where NUM_CHILDREN is an integer telling how many children to run in parallel. 

The get_children command tells you the current max number of undermine processes. 

Usage: 

bin/remote_commit [-u <USER>] get_children 
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4.1. 8.2.4 Building and clobbering your SUT (build, clobber, rclobber) 

The build and clobber commands run the _rcoverride_build and 

_rcoverride_clobber functions you provide in your rcover rides . sh file; in other words, they 
locally build and clobber your SUT. The rclobber command runs the _rcover ride_clobber 
function, but on the remote testing environment. 

Usage: 

bin/remote_commit [-u <USER>] build 
bin/remote_commit [-u <USER>] clobber 
bin/remote_commit [-u <USER>] rclobber 

4.1.8.2.5 Seeing results (summary) 

In addition to viewing results of tests via the overview GUI, you can also use remote commit's summary 
command, which prints out a summary of your remote testing environment's output directory. If 
provided, the _rcover ride_summary function is also run after printing the default summary 
information. 

Usage: 

bin/remote_commit [-u <USER>] summary 

4.1. 8.2. 6 Other commands (client) 

- u <USER>The client command runs an arbitrary command with the overmind client 
(bin/overmind_admin) on the remote overmind instance. 

Usage: 

bin/remote_commit [-u <USER>] client <CMD> 
where <CMD> is the command you want to run. See bin/overmind_admin - h for a list of potential 
commands to run. 

4.1.9 Automatically Generating Plans 

Sometimes, it’s useful to run a test script with certain types of assets and certain parameters without 
having to write a plan file. To support this, the autoplan tool is provided. This tool allows you to 
automatically generate test plans based on command line parameters. To use autoplan via remote 
commit, the subcommands autoparse, autosolve, autosubmit, autorun, and autorunlite are exposed. 
These commands are analogous to the normal parse, solve, submit, run, and runlite commands, except 
that they use an autogenerated plan instead of a pre-written one. 

To use autoplan, you specify the name of a test script to run (either filesystem path or python import 
notation), filters to describe what kind of asset is needed for each hostslot the script accepts, and 
potential values for each parameter slot the script accepts. Optionally, plan processing arguments (such 
as samples, n_notes, etc) may be specified as keyword arguments. 

For example suppose you have a test script called “mywidget.command_test”. Suppose this test script 
takes three assets: one running Windows 7 Ultimate SP2 64-bit, the other running Windows XP 
Professional SP3 32-bit, and the third being any asset. Suppose also that this test script accepts two 
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arguments, and you wish to run test plans where the first argument is either “yes” or "no” and the 
second is a number from 1 to 5. Finally, suppose you wish to run two replications of each test case. To 
automatically generate a test plan according to these specifications and then see it parsed to ensure it 
does what you want, you could run the following command: 

bin/remote_commit autoparse mywidget . command_test -H 
"os= , 7ult',ossp=sp2, arch=x64" -H os=xppro, ossp=sp3 -H " 

-p yes, no -p 1,2, 3, 4, 5 replications=2 
If you wanted to actually see what resources it would use, you could run the same command, but with 
the autosolve subcommand instead of autoparse. If you wanted to submit this autogenerated 
plan, without purging any currently running tests in your remote commit namespace, you could run the 
same command, but with autosubmit instead of autoparse. 

To submit the autogenerated plan in the normal method (where any currently running tests in your 
remote commit userspace are purged and your new plan is submitted), use the autorun or autorunlite 
subcommands instead of autoparse. Like the normal run and runlite, run will perform a clobber and 
build on the local side before syncing to the testing server, whereas runlite will not. 

Note also that you can parse and solve automatically generated plans locally without having to contact 
the testing server. To do this, instead of running... 

bin/remote_commit autoparse ... 

...run this command... 

bin/autoplan parse ... 

(and to do a local solve, use “solve” instead of “parse”). 

Autoplan does support a few other lesser-used subcommands than those listed here. See 
bin/autoplan -h (run from tyworkflow) for more information. 

4.1.9.1 Quoting with Autoplan 

Due to the interaction of the shell and the tyrant commandline parser, certain strings in hostslots or 
parameter slots unfortunately must be quoted in special ways. If a host slot held value or a parameter 
slot value contain any characters other than letters, numbers, or underscores, or if one of these values 
begins with numbers, those values must be quoted. Furthermore, since the shell normally strips quotes, 
the overall hostslot or parameter slot argument must be double-quoted, or a single set of quotes must 
be escaped. 

For example, suppose you have a test script which requires XP Professional assets. You would specify a 
host slot as follows: 

-H os=xppro 

Nothing too unusual here. If, however, you want to run a test script with 7 Professional assets (i.e. assets 
whose “os” held equals "7pro”), you must specify the host slot argument in one of the following ways: 

-H os=V7proV 
-H os=\"7pro\" 

-H "os='7pro"' 

-H 'os="7pro'" 
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In the first two cases, the sets of quotes are escaped so that the shell will not strip them. In the second 
two cases, the string is double-quoted. The shell will strip the outer set of quotes, but the inner set will 
make it into the tyrant commandline parser intact. 

These rules hold for other scenarios, such as: 

• a computer name with spaces and commas: 

-H "computer^ complicated, computer name'" 

• a pool name with a dash 

-H pool=Vdash-poolV 

• a parameter slot value with spaces and dashes: 

-p "simpleval, 'com-plex val'" 


45 


UNCLASSIFIED 



UNCLASSIFIED 


5 Appendix A - Event Detection 

Via the magnum add-on repository, Tyrant provides the ability to run arbitrary test scripts in a wrapper 
which monitors a test resource’s screen for changes. 

Magnum provides two methods of acquiring screenshots: using native Windows functionality to take 
screenshots (which only works for Windows resources and can be affected by conditions on the resource 
being tested, but can work on VMs and physical machines alike) and using ESXi screenshot functionality 
(which only works for VMs, but works for all OS families and is unaffected by conditions on the resource 
being tested). Here, we cover how to setup and verify ESXi-based event detection. 

This appending assumes that the test range you’re using has already been set up for event detection. 

5.1 Event Detection Theory 

Event detection works by taking screenshots according to a configurable interval and comparing them to 
find differences. When differences are found, they are analyzed to determine whether they are 
considered significant or not. A difference is significant if the number of changed pixels in a set of 
predefined areas of interest exceeds a defined threshold. The areas of interest and the threshold were 

determined empirically and are defined in src/magnum/event_detectors/ init . py as 

percentage boxes bounding the areas of interest. The current threshold is 10000 pixels, and the current 
areas of interest are: 

• A box in the bottom right corner of the screen, extending 25% toward the left and 50% toward 
the top. 

• A box in the center of the screen, whose edges are all 30% away from their respective screen 
edge (i.e. the left edge of the box is 30% from the left edge of the screen, the bottom edge of the 
box is 30% from the bottom edge of the screen, etc). 

• A box in the upper right corner extending 20% toward the left and 20% toward the bottom. 

When no problems occur with event detection, the event detection harness simply returns the result 
returned by the underlying test script. If, however, the test script returns SUCCESS, but any problems 
are encountered with event detection (e.g. not being able to take screenshots frequently enough to 
satisfy the configured interval), then the harness will return ATTENTION along with a message 
describing what happened. 

5.2 Testing in Adverse Environments 

The goal of testing is always to determine truthful outcomes to potential scenarios as early as possible, 
so that risks can be understood and evaluated. It is critical for users and testers to be aware of the fact 
that testing of any kind produces traces and artifacts. These traces and artifacts are created because it is 
impossible to actuate components that set up test preconditions without changing the state of the 
machine under test. There are many ways to achieve these actuations. Different methods can (and 
sometimes do) provide different results. This is especially true in adverse environments. For example: a 
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tool that passes when a user runs the program from the desktop with the mouse could fail or cause a 
pop-up if started by another process. 

Automated testing frameworks like DART allow testers to cover a greater number of potential scenarios 
than they could manually. This creates more confidence in the tools being tested. However, it is critical 
to understand that the automated framework runs in a formulaic way, so it is possible that the methods 
chosen could routinely produce different results in the real world. It is even possible that by allowing the 
automated framework to run in an adverse environment, that environment will be changed enough that 
a different result could show up. 

Tester note: You should run a statistically relevant subset of the tests by hand to verify the results given 
by the automated framework. Follow the spirit of the test plan and ensure that doing things manually 
produces the same results. This will nearly always be the case, but we have observed instances where 
there are slight deviations in the past. 

5.3 Environment Setup 

In order to use event detection, you need to do some setup in your local testing environment (the 
environment in which you run remote_commit to submit tests to the range). 

• In the same directory where your tyworkflow and tybase clones are, clone the magnum 
repository (hg clone http : //testserver . example . com : 8000/magnum). 

• In that same directory, also clone the provided PIL (Python Imaging Library) repository matching 
the architecture of the test server. For example, if your test server is running 32-bit Linux, 
choose the “PIL-linux-i686” repository, but if it’s running 64-bit Linux, use “PIL-linux-x86_64”. 

This library is used to compare screenshots to find changes. 

• In your magnum clone, copy conf ig/main . conf . example to conf ig/main . conf and 
set the following settings. Magnum has many features besides event detection, so some of 
these settings are unrelated but need to have some value set for them to prevent warning 
messages. 

o Set tester/ evdet_type to esxi. 

o Set tester/esxi_evdet_ds_name to the name of the NFS datastore you created 
in the previous section (e.g. tyrantshare). 

o Set tester/esxi_evdet_local_ds to the path on the test server of the directory 
what was exported via NFS in the previous section (e.g. 

/proj /testing/tyrant share), 
o Set server/upd_root and server/inst_root both to /tmp. 
o Set server/ip_addr to 127.0.0.1. 
o Set repository/repo_path, repository/repo_url, 

repository/repo_user, and repository/repo_pass to UNUSED. 
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5.4 Usage 

To use event detection, you use an event detection harness leafnode provided by magnum. You tell it 
what test script you want it to run and what arguments and keyword arguments to run it with and give it 
various other pieces of information to configure the event detection. The harness takes the following 
arguments: 

• host_index: Zero-counting integer index of which host event detection should be performed 
on. This is necessary when your test case uses more than one host, but you want event 
detection run on some host other than the first. Default is to use the first host. 

• interval: Float number of seconds for the screen polling interval. Default is 5 seconds. 

• use_emissary: Boolean indicating whether or not to use emissary when using the onhost 
event detection method. Not applicable for esxi event detection. Default is False. 

• type: Selects which type of event detection to perform (esxi or onhost). Default is 
onhost. For what this appendix is covering, you will always choose esxi here. 

• esxi_host: For esxi event detection, specifies the ESXi server to connect to to take 
screenshots. This may be either the specific ESXi host the VM resides on, or (we assume, but 
have not tested) a vCenter server for the range in which the VM resides (we assume this because 
other operations like reverting work both when directly connected to an ESXi host or when 
connected to a vCenter server). If not specified, the harness will attempt to query the overmind 
database (if present) and will assume the reaper field for the test resource indicates the DNS 
name or IP address of the ESXi host the VM resides on. 

• esxi_user: Username to use for connecting to the ESXi host. 

• esxi_pass: Password to use for connecting to the ESXi host. 

• vm_name: ESXi name of the VM. If not specified, the harness will attempt to query the 
overmind database (if present) and will use the computer name field as the VM name in ESXi. 

• debug: Boolean indicating whether or not to perform event detection debugging. This causes 
the generation of extra log messages and several intermediate images during the screen 
differencing process. Do not use this unless you actually need to debug the screen differencing 
process. This does not affect debugging for the test script being wrapped. Default is False. 

• debug_dir: If debug is True, specifies a directory path to which to output the extra files 
generated by debugging. Default is to use a subdirectory of the output directory for the run of 
the harness. 

• keep: Boolean indicating whether or not to keep screenshots which indicate differences 
determined to be insignificant. Default is False. 

• test: Specification of the leafnode to run. Specify this as a “python import path”, not a 
filesystem path. 

• test_args: List of positional arguments to the specified leafnode. 

• test_kwargs: Diet of keyword arguments to the specified leafnode. 

Note that these arguments are subject to the argument quoting rules of undermine. See the output of 
bin/undermine -h (run in tybase) for details. 
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5.4.1 With Undermine 

To use event detection to run single test instances with undermine, simply call the event detection 
harness with the proper arguments. For example: 

bin/undermine magnum . harness . evdet_harness 192.168.56.101 
192.168.56.103 192.168.56.104 -- host_index=@l interval=@3 . 0 
type=esxi esxi_host=192 . 168 . 56 . 10 esxi_user=someuser 
esxi_pass=somepass vm_name=test_vm_0Ol 
test=mysut . tests . sometest test_args=@"[yes, yes, no]" 
test_kwargs=@"{one=l, two=2}" 

If running out of an environment linked to an overmind range (i.e. tybase and tyworkflow are linked and 
tyworkflow’s db . rc is configured to use an overmind database) and using IP addresses that are part of 
the range, you can leave out the esxi_host and vm_name arguments and they will be determined 
automatically by looking in the overmind database. 

5.4.2 With Overmind 

To use event detection in an overmind range, for each test script you wish to run with event detection, 
you’ll need to write your own test plan that calls the event detection harness with the arguments as 
explained previously. 

To make this easier, start with the following template plan: 

from tyworkflow. support . planlang import * 
test = TESTCASE( 

script = 'magnum. harness. evdet_harness', 
hostslots = [PUT_H0STS_HERE] , 
samples = -1, 
namespace='TEST_NAME-$t ' , 
paramslots = [ 

[ ' test=LEAFNODE_SPEC' ] , 

[ ' esxi_user=USERNAME ' ] , 

[ ' esxi_pass=PASSWORD ' ] , 

[ ' keep=@True' ] 

] 

) 

EXECUTE( 

testcase = test, 

) 

Save this code to a new tile with your other plans and scripts (DO NOT simply modify this tile in place), 
and modify the copy as follows: 

• In the hostslots parameter, replace PUT_H0STS_HERE with HOST objects according to the 
number and type of hosts your test script requires. (If your test script takes multiple hosts and 
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the host on which you want event detection is not the first host, remember to set the 
host_index keyword argument in paramslots). 

• If you want to have a default limit on the number of samples the plan will generate, then change 
the samples parameter. You probably just want to leave it at - 1, though. 

• In the namespace parameter, replace TEST_NAME with a very concise name of your test; this 
will be used as part of the default namespace name when submitting this plan. 

• In paramslots, 

0 Replace LEAFNODE_SPEC with the specification of your leafnode (using “python 
import path”, as before). 

0 Replace USERNAME and PASSWORD with the username and password used to log in to 
the ESXi hosts on your range. 

0 If you have a central vCenter server managing all the ESXi hosts in the range, you can 
define add the esxi_host parameter in paramslots with the vCenter server’s DNS- 
resolvable hostname or IP address as the value. Otherwise, leave esxi_host out and 
it will be determined by looking in the overmind database. 

o Add definitions for test_args (a list) and test_kwargs (a diet) to paramslots to 
define the positional and keyword arguments to be passed to your test script being run 
under event detection. 

At this point, you should now have a working test plan that will run your test script under event 
detection. You can run this test plan with remote commit like any other, and see its results in overview, 
except now, you will get screenshots as well. 
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6 Appendix B - Detailed Repository Layouts 

1 tybase 

• bin: Shell scripts used to run Tyrant components present in tybase 

• docs: Some documentation which is superseded by this manual 

• leafbags: Directory in which collections of test scripts and plans are linked in, making them 
available to be run by undermine or overmind 

• media: Directory in which third-party supporting media is included or linked in 

0 lib_esxi-0.1: library which allows controlling ESXi servers via the vSphere API 

• PythonLocal: Built-in python distribution used by Tyrant. This directory is only present after 
running make. 

• rc: Configuration files for components in tybase. 

0 defaults: Default settings which are checked into the repository; these are overridden by 
settings in the files directly in rc. 

• src: Source code for Tyrant components in tybase. This directory is present on the python path 
when running any Tyrant components. 

0 leafbag: A collection of built-in leafnodes. 

0 tybase 

■ palantir: Source code for the palantir component, used to run operations on test 
resources 

• installer: The code used to install palantir on test resources, as well as a 
pre-built python for the OS/architecture combinations supported by 
tybase. 

■ support: Supporting modules used by various parts of Tyrant. 

■ undermine: Source code for parsing and running leafnodes. 

• test: A collection of regression tests for tybase components. 

2 tyworkflow 

The tyworkflow repository is structured similarly to tybase. The list below highlights the differences. 

• install: Code used to install overmind and reaper as a system service. 

• src 

0 leafbag: Built-in leafbag containing test scripts and plans for verifying a range is properly 
set up. 

0 tyworkflow 

■ overmind: Source code for overmind, the component which schedules tests 
across shared resources. 

■ overview: Source code for the web gui used to see test results and manage a 
range. 

■ overview_httpd: Built-in web server for running overview in small environments 
(e.g. on a laptop while traveling). 
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reaper: Source code for the reaper component, which handles sanitizing 
resources prior to a test. 

resource_manager: Source code for the component which tracks the state of 
test resources using a database. 

support: Supporting code for various components of tyworkflow. 


magnum 

• config: MAGNUM configuration files. 

0 defaults: Default settings for MAGNUM; overridden by the files directly in config. 

• src 

0 magnum 

■ event_detectors: Modules implementing the two types of event detection. 

■ harness: A leafnode which can be used to wrap your own leafnode with event 
detection logic, as well as some code for testing event detection. 

3 PIL-* 

These repositories each contain a PIL subdirectory in which reside the Python Imaging Library code and 
compiled components. 
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7 Appendix C - Commands and Usage 

All of these usage statements can be found by typing command -h from the command line. 

4 Tyworkflow 

7.1.1 remote_commit 

usage: bin/remote_commit [-u user] [ - h | help [cmd]] [start | stop | 
sync[lite] | diff | build | clobber | get_port | run[lite] [plan] * | 
parse [plan] * | solve [plan] * | submit [plan] * | get_children | 
set_children # | summary | client] 

<bin/remote_commit args>=[help | build | parse [plan] | solve [plan] | 
submit [plan] | start | stop | get_children | 
set_children # | clean | clobber | summary | get_pids] * 

7.1.2 db_admin 

Help on class db_admin in module main : 


class db_admin 

| Usage: db_admin [<option>]* <method> [arg]* 

I 

| Options: 

I -h 

| -c <rc_file> 

I -v 

(logging . DEBUG) 

I -q 

(logging .WARN) 

I -J 

I 

| Methods defined here: 

I 

| add_attr (self , attr_table, attr_name) from 
tyworkflow. resource_manager . client_util . DB 


Print help. 

Use rc_file as an alternate config file. 
Turn on verbose diagnostic output. 

Turn off verbose diagnostic output. 

return JSON-encoded output 
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| Adds a formal attribute of the given name attr_name (string) 

to the 

| given table attr_table (string). 

I 

| add_computer (self , computer, *args, **kwargs) from 
tyworkflow. resource_manager . client_util . DB 

| Adds a computer of the given name computer (string) with the 

| attributes as specified in *args/**kwargs . Valid attributes 

and their order if 

| specified positionally are as follows: 

| formal attributes: 

| ip: (str) IP address of primary interface 

| mac: (str) MAC address of primary interface 

| hwtype: (str) name of hardware type of computer 

| pool: (str) name of pool in which to place computer 

| vlan: (str) name of vlan in which to place computer 

| reaper: (str) name of reaper to use to reap computer 

I 

| Returns: the ID of the inserted computer entry 

I 

| add_recipe(self , recipe, *args, **kwargs) from 
tyworkflow. resource_manager . client_util . DB 

| Adds a recipe of the given name recipe (string) with the 

attributes 

| as specified in *args/**kwargs . Valid attributes and their 

order if specified 

| positionally are as follows: 

| formal attributes: 

| family: (str) OS family enumeration (e.g. 'win 1 , 

1 linux ' ) 
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| os: (str) OS name (e.g. 'xp', 'vista', 'fedora') 

| ossp: (str) OS service pack designation 

| lang: (str) OS language enumeration (e.g. 'en-US') 

| arch: (str) OS architecture enumeration, typically 

either 'x86' or 'x86_64' 

| apps: (str) names of apps installed in the recipe 

I 

| Returns: the ID of the inserted recipe entry 

I 

| add_snapshot(self , computer, recipe, snapshot=None) from 
tyworkflow. resource_manager . client_util . DB 

| Adds a snapshot with a certain recipe to a computer. 

I 

| Parameters: 

| str|int computer: name or ID of the computer to add the 

snapshot to 

| str|int recipe: name or ID of the recipe of the snapshot 

being added 

| str snapshot: name of the snapshot being added, defaults 

to latest if not 

| set 

I 

| Returns: the ID of the added snapshot 

I 

| del_attr (self , attr_table, attr_name) from 
tyworkflow. resource_manager . client_util . DB 

| Deletes the attribute attr_name (string) from the table 

attr_table 

| (string). Returns the entry which was deleted, or None if the 

attribute didn't 
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| exist . 

I 

| del_computer (self , computer) from 
tyworkflow. resource_manager . client_util . DB 

| Deletes the named computer (string). Returns the record of 

| the deleted computer, or None if the given computer didn't 

exist . 

I 

| del_recipe(self , recipe) from 
tyworkflow. resource_manager . client_util . DB 

| Deletes the named recipe (string). Returns the record of the 

deleted 

| recipe, or None if no such recipe existed. 

I 

| del_snapshot(self , computer, recipe=None) from 
tyworkflow. resource_manager . client_util . DB 

| Deletes all snapshots from the named computer (string). If 

recipe 

| (string) is specified, then only snapshots with that recipe 

will be 

| deleted. Returns the record(s) for the deleted snapshot(s) or 

None if either 

| the given computer or recipe don't exist. 

I 

| describe_table(self , table) from 
tyworkflow. resource_manager . client_util . DB 

| Returns a string description of the table named table 

(string) . 


| drop_db(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Drop all tyrant tables and entries including test results. 
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| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| drop_resources(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Drop the tyrant resource tables and entries (computer, recipe, 

snapshot) . 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| drop_tests(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Drop the tyrant test result tables and entries. 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| dump_table(self , table) from 
tyworkflow. resource_manager . client_util . DB 

| Returns all the records in the given table (string). 

I 

| dump_version(self ) from tyworkflow. resource_manager . client_util . DB 

| Returns the version of the database. 
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I 

| export_computers(self , csvfile='-') from 
tyworkflow. resource_manager . client_util . DB 

| Export computers to csv file (or stdout), format: 

| <name>, <ip>, <mac>, <hwtype>, <pool>, <vlan>, <reaper> 

| @snapshot, <recipe>, <snapshot> 

| If the csv file is (the default), write to stdin. 

I 

| export_recipes(self , csvfile='-') from 
tyworkflow. resource_manager . client_util . DB 

| Export recipes to csv file (or stdout), format: 

| <name>, <family>, <os>, <ossp>, <lang>, <arch>, <apps> 

| If the csv file is (the default), write to stdin. 

I 

| export_snapshots(self , csvfile='-') from 
tyworkflow. resource_manager . client_util . DB 

| Export snapshots to csv file (or stdout), format: 

| <computer | id>, crecipe | id>, <snapshot | id> 

| If the csv file is (the default), write to stdin. 

I 

| get_attr_id(self , attr_table, attr_name) from 
tyworkflow. resource_manager . client_util . DB 

| Returns the ID for an attribute. 

I 

| Parameters: 

| str attr_table: name of the table to look for the 

attribute in 

| str attr_name: name (or possible ID [but still must be 

string, not an int] 
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| of the attribute to look for 

I 

| Returns: the ID of the attribute found 

I 

| import_computers(self , csvfile='-', testing_use= ' N 1 , 
testing_dirty= ' Y ' , **kwargs) from 
tyworkflow. resource_manager . client_util . DB 

| Import computers from csv file (or stdin), format: 

| <name>, <ip>, <mac>, <hwtype>, <pool>, <vlan>, <reaper> 

| @snapshot, <recipe>, <snapshot> 

| If the csv file is (the default), read from stdout. 

I 

| Parameters: 

| str csvfile: path to CSV file to import; if read from 

standard 

| input instead 

| enum(Y|N) testing_use: value to set testing_use flag to on 

| newly-imported computers; default is to mark computer 

not to be 

| used for testing 

| enum(Y|N|R) testing_dirty : value to set testing_dirty flag 

to on 

| newly-imported computers; default is to mark computer 

as dirty 

| **kwargs: keyword arguments to override values of some 

computer fields 

I 

| Fields are defined in the list_computers method's help. The 

kwargs can 

| define default values for all the formal attributes except id. 
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| import_recipes(self , csvfile='-', **kwargs) from 
tyworkflow. resource_manager . client_util . DB 

| Import recipes from csv file (or stdin), format: 

| <name>, <family>, <os>, <ossp>, <lang>, <arch>, <apps> 

| If the csv file is (the default), write to stdout. 

| The kwargs can define default values, for example: lang=en 

arch=x86 


| import_snapshots(self , csvfile='-', **kwargs) from 
tyworkflow. resource_manager . client_util . DB 

| Import snapshots from csv file (or stdin), format: 

| <computer | id>, crecipe | id>, <snapshot | id> 

| If the csv file is (the default), write to stdout. 

| The kwargs can define default values for the following 

parameters : 

I 

| str computer: name of the computer the snapshot is being added 

for 

| str recipe: name of the recipe in the snapshot 

| str snapshot: name of the snapshot 

I 

| init_db(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Incoming format: [+really-do-it] [+drop] 


| Create any missing required tyrant database tables incuding 

recipes, computers, 

| and test results. Optionally, if the argument list includes 

the string 
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| "+drop" the database is removed prior to the creation of new 

tables. The argument 

| list must include the magic string "+really-do-it" to help 

prevent accidental 

| execution of this command, or optionally, you will be prompted 

for confirmation 

| prior to command execution. 

I 

| list_attr(self , attr_table, with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists attributes of rows from the given table, with the set of 

returned 

| attributes modified by the given parameters. Ignore the 

limit_to 

| parameter below. 

I 

| Parameters: 

| str attr_table: name of the table to list attributes from 

| bool with_header: If True, a header giving the names of 

the fields will be 

| output with the returned entries. 

| str filter_by: A list in the form of a comma-separated 

string of filtering 

| rules. See the FILTERING section. Fields which may be 

filtered on are 

| listed in this object's docstring. 

| str sort_by: A list in the form of a comma-separated 

string of sorting 

| rules. Sorting rules are simply names of fields, 

optionally prepended 

| by 1 ! 1 to indicate a reverse sort on that field. 

Default sort order is 
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| ascending. 

| str|int limit_to: Limit the return from the database to 

the given number of 

| records. This limit is applied at query time. NOTE: 

Not all list 

| functions support this parameter. Check the parameter 

list to see if it 

| applies. 

| str|int display_num: First index number (zero-counting) of 

rows to return. Only 

| those rows on or after the given index number will be 

returned. Useful for 

| paging results, perhaps. 

| int cut_to: Truncates each row of results to the specified 

number of fields. 

| str select: A list in the form of a comma-separated string 

of field names 

| to select (as with the SQL SELECT clause). Only those 

fields will be 

| returned. Valid fields are the same fields which may 

be filtered, and 

| are listed in this object's docstring. 


| Examples: 

| list all resources running windows 

| bin/db_admin list_resources filter_by="family=win" 

| list all computers, sorted by ip, ascending 

| bin/db_admin list_computers sort_by=ip 

| list the ten recipes starting with recipe 2, sorted 

descending by os 

| bin/db_admin list_recipes display_num=2 

limit_to=10 sort_by=" ! os" 
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| see only the ip and mac addresses for all computers, 

without a header row 

| bin/db_admin list_computers select=ip, mac 

with_header=false 

I 

| FILTERING 

| Filters are specified using one of the following operators: 

| a ~= b: True if str a is matched by regex b 

| a == b: True if a equals b 

| a <> b: True if a does not equal b 

| a != b: True if a does not equal b 

| a >= b: True if int(a) is greater than or equal to int(b) 

| a <= b: True if int(a) is less than or equal to int(b) 

| a > b: True if int(a) is greater than int(b) 

| a < b: True if int(b) is less than int(b) 

| a = b: True if a equals b 

I 

| Examples: 

| for listing resources, match all resources with service 

pack 2 or greater 

| ossp >= 2 

| for listing computers, list the computer with ip address 

127.0.0.1 

| ip == 127.0.0.1 

| for listing test namespaces, get any namespaces starting 

with "jdoe" 

| name ~= A jdoe 


UNCLASSIFIED 


63 



UNCLASSIFIED 


| list_computers(self , with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists computers according to the given parameters. 

| See list_attr help for detailed parameter descriptions. Valid 

fields to 

| select/filter are: 

| formal attributes: 

| id: (int) ID of this thing 

| name: (str) name of this thing 

| ip: (str) IP address of primary interface 

| mac: (str) MAC address of primary interface 

| hwtype: (str) name of hardware type of computer 

| pool: (str) name of pool in which to place computer 

| vlan: (str) name of vlan in which to place computer 

| reaper: (str) name of reaper to use to reap computer 

| metadata attributes: 

| testing_use: enum(Y,N) whether computer can be 

scheduled for testing 

| testing_in_use : enum(Y,N) whether computer is 

currently in use on a test 

| testing_dirty : enum(Y,N,R) whether computer is dirty 

(Y), clean (N), or should be reaped (R) 

| reserve_name : (str) name given when 

reserving/scheduling the computer 

| reserve_time : (datetime) time at which machine was 

reserved/scheduled 

I 

| list_namespaces(self , with_header=False, filter_by=None, 
sort_by=None, limit_to=None, display_num=None, cut_to=None, 
select=None) from tyworkflow. resource_manager . client_util . DB 
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| Lists namespaces. See list_attr for detailed parameter 

| descriptions. Valid fields to select/filter are: 

| formal attributes: 

| id: (int) ID of this thing 

| name: (str) name of this thing 

| nid: (int) id of the namespace 

| n_name: (str) name of the namespace 

| start_time: (datetime) time namespace started running 

tests 

| end_time: (datetime) time namespace finished running 

all tests 

| combos_total: (int) number of combos in namespace 

| success: (int) number of combos with success status 

| failure: (int) number of combos with failure status 

| attention: (int) number of combos with attention 

status 

| skipped: (int) number of combos with skipped status 

| error: (int) number of combos with error status 

| purged: (int) number of combos with purged status 

| running: (int) number of combos with running status 

| pending: (int) number of combos with pending status 

| n_notes: (str) notes for the namespace 

I 

| list_recipes(self , with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists recipes. See list_attr for detailed parameter 

| descriptions. Valid fields to select/filter are: 

| formal attributes: 
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| id: (int) ID of this thing 

| name: (str) name of this thing 

| family: (str) OS family enumeration (e.g. 'win 1 , 

1 linux' ) 

| os: (str) OS name (e.g. 'xp', 'vista 1 , 'fedora') 

| ossp: (str) OS service pack designation 

| lang: (str) OS language enumeration (e.g. 'en-US') 

| arch: (str) OS architecture enumeration, typically 

either 'x86' or 'x86_64' 

| apps: (str) names of apps installed in the recipe 


| list_resources(self , with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists resources. See list_attr for detailed parameter 

| descriptions. Valid fields to select/filter are any of the 

fields for 

| computer, recipe or snapshot (see the corresponding list_* 

methods for 

| the lists of valid fields), except that the "id" fields for 

each are 

| called "computer_id", "recipe_id" and "snapshot_id" , 

respectively, and 

| the "name" fields are called "computer", "recipe", and 

"snapshot", 

| respectively. 


| list_snapshots(self , with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists snapshots. See list_attr for detailed parameter 

| descriptions. Valid fields to select/filter are: 
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| formal attributes: 

| id: (int) ID of this thing 

| name: (str) name of this thing 

| computer: (str) name of computer the snapshot is on, 

or specify as computer_id and integer ID of computer 

| recipe: (str) name of recipe on the snapshot, or 

specify as recipe_id and integer ID of recipe 

| snapshot: (str) name of snapshot 

| metadata attributes: 

| testing_fubar : enum(Y,N) whether or not the snapshot 

is broken 

I 

| list_tables(self , with_header=False) from 
tyworkflow. resource_manager . client_util . DB 

| Lists tables in the database. If with_header is True, a header 

| will be returned with the results (which in this case is just 

the string 'name' 

| since the results are single-field rows of table names). 

I 

| list_testcase_files(self , with_header=False, filter_by=None, 
sort_by=None, display_num=None, cut_to=None, select=None) from 
tyworkflow. resource_manager . client_util . DB 

| Lists files in all the testcases. See list_attr for detailed 

| parameter descriptions. Valid fields to select/filter on are: 

| formal attributes: 

| tid: (int) id of the testcase the file is for 

| id: (int) id of the file 

| output_path: (str) path to the file 

| In addition, a "name" field (str) may be used only for 

filtering on, 
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| which sometimes contains a relative path to the file. 

I 

| list_testcases(self , with_header=False, filter_by=None, 
sort_by=None, limit_to=None, display_num=None, cut_to=10, select=None) 
from tyworkflow. resource_manager . client_util . DB 

| Lists testcases. See list_attr help for detailed parameter 

| descriptions. The limit_to parameter is valid for this method. 

Valid 

| fields to select/filter on are: 

| formal attributes: 

| tid: (int) id of the testcase 

| t_name: (str) name of the testcase 

| pid: (int) id of the testplan the testcase is in 

| p_name: (str) name of the testplan the testcase is in 

| nid: (int) id of the namespace the testcase is in 

| n_name: (str) name of the namespace the testcase is in 

| s_name: (str) name of the script the testcase is 

running 

| metadata attributes: 

| start_time: (datetime) time testcase started running 

| end_time: (datetime) time testcase ended running 

| result_code: (str) result code testcase ended with 

| result: (str) result value testcase ended with 

| o_path: (str) path of output directory for testcase 

I 

| list_testplans(self , with_header=False, filter_by=None, 
sort_by=None, limit_to=None, display_num=None, cut_to=None, 
select=None) from tyworkflow. resource_manager . client_util . DB 

| Lists testplans. See list_attr help for detailed parameter 
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I 

Valid 

I 

I 

I 

I 

I 

I 

I 

running 

I 

I 

I 

I 

I 

I 

I 


descriptions. The limit_to parameter is valid for this method 

fields to select/filter on are: 
formal attributes: 

pid: (int) id of the testplan 

p_name: (str) name of the testplan 

nid: (int) id of the namespace the testplan is in 

n_name: (str) name of namespace the testplan is in 

s_name: (str) name of the script the testplan is 

metadata attributes: 

start_time: (datetime) time testplan started running 
end_time: (datetime) time testplan ended running 
combos_total : (int) number of combos in testplan 
success: (int) number of combos with success status 
failure: (int) number of combos with failure status 
attention: (int) number of combos with attention 


status 

| skipped: (int) number of combos with skipped status 

| error: (int) number of combos with error status 

| purged: (int) number of combos with purged status 

| running: (int) number of combos with running status 

| pending: (int) number of combos with pending status 

| p_notes: (str) notes on the testplan 

I 

| migrate_db(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Migrate tyrant database from previous version. 
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| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| purge_computers(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Purge the tyrant computer, snapshot, and computer attr table 

entries . 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| purge_recipes(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Purge the tyrant recipe, snapshot, and recipe attr table 

entries . 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| purge_resources(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Purge the tyrant resource table entries (computer, recipe, 

snapshot) . 

| The argument list must include the magic string "+really-do- 

it" to help prevent 
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| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| purge_snapshots(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Purge the tyrant snapshot and snapshot attr table entries. 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

I purge_tests(self , *args) from 
tyworkflow. resource_manager . client_util . DB 

| Purge the tyrant test result table entries. 

| The argument list must include the magic string "+really-do- 

it" to help prevent 

| accidental execution of this command, or optionally, you will 

be prompted for 

| confirmation prior to command execution. 

I 

| query(self, *args) from tyworkflow. resource_manager . client_util . DB 

| Runs the given query against the database and returns all 

results . 

| *args is a list of strings which will be joined on spaces to 

form the query to 

| be executed. 

I 

| reserve_asset(self , reserve_name= ' NULL 1 , **attrs) from 
tyworkflow. resource_manager . client_util . DB 
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| Reserves assets (computers). 

I 

| Parameters: 

| str reserve_name : name to store on the computer record to 

identify who ' s 

| reserved it 

| **attrs: keyword args specifying values to match on 

computer fields. 

| Keys are the names of valid computer fields (see 

list_computers), 

| values are what those fields must equal. Only matching 

computers 

| will be reserved. 

I 

| Examples: 

| reserve a computer with a specific id 

| bin/db_admin reserve_asset my_name id=104 

| reserve all computers in a specific pool 

| bin/db_admin reserve_asset my_name pool=pool001 

| reserve all computers not currently reserved 

| bin/db_admin reserve_asset my_name testing_use=Y 

I 

| set_computer_attr(self , attr_table, attr_name, **attrs) from 
tyworkflow. resource_manager . client_util . DB 

| Sets the given formal attribute for a computer. 

I 

| Parameters: 

| str attr_table: name of the formal attribute being set 

| str attr_name: the value of the attribute being set 
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| {str, str} **attrs: Keyword arguments specifying formal or 

metadata 

| attributes and their values. If specified, then only 

entries with 

| attributes matching the values given here will be 

updated. Otherwise, 

| all entries are updated. 

I 

| Valid keywords for **attrs are the formal attribute names 

listed in the help for 

| list_computers . 

I 

| set_computer_flag(self , flag_name, flag, **attrs) from 
tyworkflow. resource_manager . client_util . DB 

| Sets the given flag (metadata attribute) for a computer. 

I 

| Parameters: 

| str flag_name: name of the flag to set 

| str flag: Value to set for the flag. 

| {str, str} **attrs: Same as set_computer_attr, but only 

metadata 

| attributes may be set. 

I 

| Valid keywords for **attrs are those metadata attributes 

listed in 

| list_computer ' s help which are flags (e.g. take 'Y', ' N ' , or 

'R' 

| values). 

I 

| set_recipe_attr(self , attr_table, attr_name, **attrs) from 
tyworkflow. resource_manager . client_util . DB 
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| Sets the given attribute for a recipe. See set_computer_attr 

for 

| detailed parameter descriptions. Valid **attrs values are the 

| formal attribute names listed in list_recipes ' s help. 

I 

| set_resource_flag(self , flag_name, flag, **attrs) from 
tyworkflow. resource_manager . client_util . DB 

| Sets the given metadata attribute for a computer/snapshot. 

I 

| Parameters: 

| str flag_name: name of the flag to set 

| str flag: Value to set for the flag. 

| {str, str} **attrs: Keyword arguments specifying formal or 

metadata 

| attributes and their values. If specified, then only 

entries with 

| attributes matching the values given here will be 

updated. Otherwise, 

| all entries are updated. 


| Valid keywords for **attrs are the metadata attribute names 

listed in 

| list_resource ' s help which take flag values (e.g. 'Y', ' N ' , 

'R' ) ■ 


| Examples: 

| clear fubar flag on all resources in pooll: 

| bin/db_admin set_resource_flag testing_fubar ' N ' 

pool=pooll 

| reserve computer with id 100: 
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| bin/db_admin set_resource_flag testing_use ' N ' 

computer_id=100 

I 

| set_snapshot_flag(self , flag_name, flag, **attrs) from 
tyworkflow. resource_manager . client_util . DB 

| Sets the given flag (metadata attribute) for a snapshot. See 

| set_computer_flag for detailed parameter descriptions. Valid 

keywords 

| for **attrs are the metadata attributes listed in 

list_snapshot ' s help 

| which take flag values (e.g. 'Y 1 or 'N'). 

I 

| unreserve_asset(self , **attrs) from 
tyworkflow. resource_manager . client_util . DB 

| Unreserves assets, making them available for testing. **attrs 

is the 

| same as for reserve_asset . 

I 

| Examples: 

| unreserve all computers reserved with a specific name 

| bin/db_admin unreserve_asset reserve_name=my_name 

| unreserve all reserved assets 

| bin/db_admin unreserve_asset testing_use=N 

| unconditionally unreserve everything 

| bin/db_admin unreserve_asset 


7.1.3 overmind_admin 

Help on class client in module tybase . support . client : 

75 


UNCLASSIFIED 


UNCLASSIFIED 


class client 

| Usage: client. py [option]* <cmd> [<arg>]* 


Options : 


-h 

-v 

-q 

-t <timeout> 
-s <server> 
-p <port> 


Print help. 

Enable verbose mode, (logging . DEBUG) 
Disable verbose mode, (logging .WARN) 
Set command timeout value. 

Server to connect to. 

Port to connect to. 


Methods defined here: 


I purge_plan(self , nid='', pid='', tid= 1 ' ) from 
tyworkflow. overmind . commands . Commands 

| Purge the plan, match on attributes (all integers) 

| n(amespace)id 

| p(lan)id 

| t(est)id 


| service_echo(self , *args, **kargs) from 
tyworkflow. overmind . commands . Commands 

| Return arguments. 


| service_get_children(self ) from 
tyworkflow. overmind . commands . Commands 
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| Retrieves the current maximum number of child processes in the 

service . 

I 

| service_log_level(self , log_level, *logs) from 
tyworkf low. overmind . commands . Commands 

| Call setLogLevel on the root logger and any other named 

loggers 

I 

| service_ping(self ) from tyworkflow. overmind . commands . Commands 
| Return True. 

I 

| service_set_children(self , max_children) from 
tyworkflow. overmind . commands . Commands 

| Set the integer maximum number of child processes in the 

service . 

I 

| service_shutdown(self ) from tyworkflow. overmind . commands . Commands 
| Shut the server down. 

I 

| service_timestamp(self ) from tyworkflow. overmind . commands . Commands 
| Return service start time. 

I 

| service_uptime(self ) from tyworkflow. overmind . commands . Commands 
| Return service uptime. 

I 

| submit_plan(self , planmod, **plan_opts) from 
tyworkf low. overmind . commands . Commands 

| Submit the plan specified in the string planmod. 

| Keyword arguments override plan attributes of the same name. 
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7.1.4 overmind 

Help on class overmind in module main. 

class overmind 

| Usage: overmind [<option>]* 


| Options: 


1 -h 

Print help. 

| -c <rc_file> 

Use rc_file as an alternate config file. 

| -1 <host> 

Listen for connection on host ip. 

| -p <port> 

Listen for connections on port. 

| -o <dir> 

Output directory to use. 

1 -f 

daemon) . 

Run overmind in the foreground (Default is as ; 

1 -b 

Run overmind in the background. 

1 -v 

Turn on verbose diagnostic output. 

1 -q 

Turn off verbose diagnostic output. 

1 -x 

Require at least one argument to start service 


| Methods defined here: 

I 

| purge_plan(self , nid='', pid= ' ' , tid='') from 
tyworkf low. overmind . commands . Commands 

| Purge the plan, match on attributes (all integers) 

| n(amespace)id 

| p(lan)id 

| t(est)id 
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I 

| service_echo(self , *args, **kargs) from 
tyworkflow. overmind . commands . Commands 

| Return arguments. 

I 

| service_get_children(self ) from 
tyworkflow. overmind . commands . Commands 

| Retrieves the current maximum number of child processes in the 

service . 

I 

| service_log_level(self , log_level, *logs) from 
tyworkflow. overmind . commands . Commands 

| Call setLogLevel on the root logger and any other named 

loggers 

I 

| service_ping(self ) from tyworkflow. overmind . commands . Commands 

| Return True. 

I 

| service_set_children(self , max_children) from 
tyworkflow. overmind . commands . Commands 

| Set the integer maximum number of child processes in the 

service . 

I 

| service_shutdown(self ) from tyworkflow. overmind . commands . Commands 

| Shut the server down. 

I 

| service_timestamp(self ) from tyworkflow. overmind . commands . Commands 

| Return service start time. 

I 

| service_uptime(self ) from tyworkflow. overmind . commands . Commands 
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| Return service uptime. 

I 

| submit_plan(self , planmod, **plan_opts) from 
tyworkf low. overmind . commands . Commands 

| Submit the plan specified in the string planmod. 

| Keyword arguments override plan attributes of the same name. 


7.1.5 reaper_admin 

Help on class client in module tybase . support . client : 


class client 

| Usage: client. py [option]* <cmd> [<arg>]* 


Options : 


-h 

-v 

-q 

-t <timeout> 
-s <server> 
-p <port> 


Print help. 

Enable verbose mode, (logging . DEBUG) 
Disable verbose mode, (logging .WARN) 
Set command timeout value. 

Server to connect to. 

Port to connect to. 


Methods defined here: 


| revert_host(self , host, snapshot='', reaper^ 1 ') from 
tyworkflow. reaper . commands . Commands 

| Revert the vm at the specified ip. 
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I 

| service_echo(self , *args, **kargs) from 
tyworkflow. reaper . commands . Commands 

| Return arguments. 

I 

| service_log_level(self , log_level, *logs) from 
tyworkflow. reaper . commands . Commands 

| Call setLogLevel on the root logger and any other named 

loggers 

I 

| service_ping(self ) from tyworkflow. reaper . commands . Commands 
| Return True. 

I 

| service_set_children(self , max_children) from 
tyworkflow. reaper . commands . Commands 

| Set the maximum number of concurrent reverts allowed. 

I 

| service_shutdown(self ) from tyworkflow. reaper . commands . Commands 
| Shut the server down. 

I 

| service_timestamp(self ) from tyworkflow. reaper . commands . Commands 
| Return service start time. 

I 

| service_uptime(self ) from tyworkflow. reaper . commands . Commands 
| Return service uptime. 
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5 Tybase 

7.1.6 palantir_admin 

Help on class client in module tybase . support . client : 


class client 

| Usage: client. py [option]* <cmd> [<arg>]* 

I 

| Options: 


I 


-h 

-v 

-q 

-t <timeout> 
-s <server> 
-p <port> 


Print help. 

Enable verbose mode, (logging . DEBUG) 
Disable verbose mode, (logging .WARN) 
Set command timeout value. 

Server to connect to. 

Port to connect to. 


I 


| Methods defined here: 

I 

| clone(self, reuse_session=False) from 
tybase . palantir . client . Client 

| Creates and opens a new Client object to 

(self . host, self . port) . 

| + reuse_session flag causes the session to be reused by 

copy. 

| + WARNING: client . close( ) will close the session on the 

for all copies. 


| createEmissary(self , *args, 
tybase . palantir . client . Client 


*kargs) from 


the 

server 
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| execcmd(self , *args, **kargs) from tybase . palantir . client . Client 
| Runs a command remotely, waiting for it complete. 


| execfunc(self , path, *args, **kargs) from 
tybase . palantir . commands . Commands 

| Return path(*args, **kargs) 

I 

| An execfunc user has implicit access to these variables: 

| self --> palantir . commands . Commands( ... 

support . netcom.ServerCommands , threading . Thread ) 

| self. server --> 

palantir . server .Server( support . netcom . Server ) 

| self . command_id --> (NEW) this command_id currently being 

processed 

I 

| An execfunc use has implicit access to these functions: 

| self . set_session_variable( ) 

| self . get_session_variable( ) 

| self . ANY_METHOD_lN_COMMANDS_SUBCLASS -- e.g. execfunc from 

palantir /commands/Commands 

I 

| execute(self , opts, args) from tybase . palantir . commands . Commands 

| Runs a command (e.g. a new process) on the remote side. 


| The elements of the command line are given as positional 

arguments in 

| *args. Specify your command-line already tokenized, as you 

would when 

| using python's subprocess module (which is what's used on the 

remote 
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side) . 


| opts and **kargs serve the same purpose. They let you specify 

keyword 

| arguments to affect how the command is run on the remote side. 

Valid 

| keywords and their meaning are as follows: 

| bool wait: whether to wait for the command to complete 

before 

| execute returns; defaults True 

| str cwd: current working directory for running the 

command; 

| defaults to the current working directory at the time 

of 

| invocation 

| str stdin: path to a file on the remote side from which to 

read 

| data to the process's standard in 

| str stdout: path to a file on the remote side to which to 

write 

| the process's standard out 

| str stderr: path to a file on the remote side to which to 

write 

| the process's standard error 

| bool detach: whether to detach the remote process from the 

palantir 

| server which invoked it; defaults True 

| bool shell: if True, invoke the command in the shell; use 

this if 

| you need to use shell pipelining or for some other 

reason have 
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| to specify the command as one big string; defaults 

False. If 

| setting shell True, you almost certainly want to 

specify your 

| command as one string rather than pre-tokenized . If 

you set 

| shell=True and specify a list of commands, behavior 

will differ 

| based on the target operating system; read the python 

subprocess 

| module documentation for more info. 

| The values in kargs are merged into opts, you can specify 

these options 

| as either elements of the opts diet, or as keyword arguments 

to execute. 


| externf unc(self , path, *args, **kargs) from 
tybase . palantir . client . Client 

| Runs a blob of python code on the remote side. 


| fappend(self , rfile, data) from tybase . palantir . client . Client 
| Appends data to remote file 


| fhash(self, fname, method= l md5' , offset=0, nbytes=0) from 
tybase . palantir . commands . Commands 

| Return hash via methods [md5|crc32] 


| flength(self , rfile) from tybase . palantir . client . Client 
| Returns the size of the remote file 
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| fread(self, fname, offset=0, nbytes=0) from 
tybase . palantir . commands . Commands 

| Return file contents. 

I 

| fwrite(self, fname, fdata, offset=0) from 
tybase . palantir . commands . Commands 

| Write file contents. 

I 

| get(self, rfile, lfile=None, mode=None, force=True, 
max_bytes=None) from tybase . palantir . client . Client 

| Get the remote file and store its contents locally. 

I 

| rfile : The remote file to get . 

| lfile : The local file that will be written. 

| mode : File permissions. 

| force : Always write file (do not test). 

| max_bytes: Number of bytes to read at once (chunk size for a 

read loop) 

I 

| Return number of bytes read. 

I 

| get_os_arch(self ) from tybase . palantir . client . Client 

| Returns the standardized CPU architecture name (e.g. x86, 

x86_64, ia32, etc) of the OS. 

I 

| get_os_family(self ) from tybase . palantir . client . Client 
| Returns the standardized OS family name (e.g. linux, windows). 

I 

| get_platform(self ) from tybase . palantir . client . Client 
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| Wrapper to remote execution of sys . platform( ) 

I 

| host_ping(self , tries=l, delay=5) from 
tybase . palantir . client . Client 

| ICMP ping the remote host 

I 

| mirrorfunc(self , *args, **kargs) from 
tybase . palantir . client . Client 

| Runs a function on the remote side and gives back a proxy to 

the 

| remote return value, allowing you to work with e.g. imported 

modules, 

| open file handles, and other objects that can't be pickled and 

sent 

| across the connection normally. 

I 

| mkdir(self, name) from tybase . palantir . client . Client 
| Make a remote directory 

I 

| ostype(self) from tybase . palantir . client . Client 

| Returns a string indicating the OS that the server is running. 

I 

| path_exists(self , path) from tybase . palantir . client . Client 

| Returns True if the given filesystem path exists on the remote 

side . 

I 

| pathsep(self ) from tybase . palantir . client . Client 
| Wrapper to remote execution of os.pathsep() 
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| put(self, lfile, rfile=None, mode=None, force=True, 
max_bytes=None, remote_check=True) from tybase . palantir . client . Client 

| Test and put the local file to remote file. 


lfile 

: The local file to 

put . 


rfile 

: The remote file that will 

be written 

mode 

: File permissions. 



force 

: Always write file 

(do not 

test) . 


| max_bytes: Number of bytes to write at once (chunk size for 

the upload loop) 

I 

| Return number of bytes written. 

I 

| remotef unc(self , func, *args, **kargs) from 
tybase . palantir . client . Client 

| Given a function object, runs it remotely and gives back the 

return 

| value. 

I 

| rget(self, src_path, tgt_path=None, **kargs) from 
tybase . palantir . client . Client 

| Get files/directories at src_path to tgt_path (using rysnc 

module) 

I 

| rmdir(self, name) from tybase . palantir . client . Client 

| Delete a remote directory 

I 

| rmfile(self, name) from tybase . palantir . client . Client 

| Delete a remote file 


UNCLASSIFIED 



UNCLASSIFIED 


I 

| rmtree(self, name) from tybase . palantir . client . Client 
| Delete a remote directory tree 

I 

| rput(self, src_path, tgt_path=None, **kargs) from 
tybase . palantir . client . Client 

| Put files/directories at src_path to tgt_path (using rysnc 

module) 

I 

| sep(self) from tybase . palantir . client . Client 
| Wrapper to remote execution of os.sep() 

I 

| service_echo(self , *args, **kargs) from 
tybase . palantir . commands . Commands 

| Return arguments. 

I 

| service_log_level(self , log_level, *logs) from 
tybase . palantir . commands . Commands 

| Call setLogLevel on the root logger and any other named 

loggers 

I 

| service_ping(self ) from tybase . palantir . commands . Commands 
| Return True. 

I 

| service_secure_cert_fname(self ) from 
tybase . palantir . commands . Commands 

| Incoming format: 

I 

| Returns the string filename for the trusted SSL 
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certificate file. 

service_shutdown(self ) from tybase . palantir . commands . Commands 
Shut the server down. 

service_timestamp(self ) from tybase . palantir . commands . Commands 
Return service start time. 

service_uptime(self ) from tybase . palantir . commands . Commands 
Return service uptime. 

service_version(self ) from tybase . palantir . commands . Commands 
Return server version. 

shutdown(self ) from tybase . palantir . commands . Commands 
Shuts down the palantir server. 

spawn(self, *args, **kargs) from tybase . palantir . client . Client 
Non-blocking execution of remote commands 

sys_executable(self ) from tybase . palantir . client . Client 
Wrapper to remote execution of sys . executable( ) 

system(self, *args, **kargs) from tybase . palantir . client . Client 
Runs a command, remotely, through a shell, waiting for it to 
complete. The command SHOULD be specified as a single string, 
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| tokenized. The command MAY be specified as multiple tokens, 

in which 

| case the tokens will be joined on a space. Therefore, if 

specified as 

| tokens, each token must be individually quoted properly in 

order for 

| system to work. 

I 

| tl;dr: specify as a single string, not tokens 


7.1.7 palantir 

Help on class palantir in module main. 

class palantir( builtin .object) 

| Usage: palantir [option]* 

I 

| Options: 


-h 

-c <rc_file> 

-L <log_file> 
-t <cert_file> 
-1 <host> 

-p <port> 

-f 

-b 


Print help. 

Use rc_file as an alternate config file. 
Write log messages to log_file 
Set trusted SSL/TLS certificate file 
Listen for connection on host ip. 

Listen for connections on port. 

Run palantir in the foreground. 

Run palantir in the background. 
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| -v : Verbose output on. 

| -q : Verbose output off. 

| -x : Require at least one argument to start service. 

| -K : Creates and registers a new SSL cert for local 

connections . 

| -S : Use secure transport mode 

I 

| Methods defined here: 

I 

| execfunc(self , path, *args, **kargs) from 
tybase . palantir . commands . Commands 

| Return path(*args, **kargs) 

I 

| An execfunc user has implicit access to these variables: 

| self --> palantir . commands . Commands( ... 

support . netcom.ServerCommands , threading . Thread ) 

| self. server --> 

palantir . server .Server( support . netcom . Server ) 

| self . command_id --> (NEW) this command_id currently being 

processed 

I 

| An execfunc use has implicit access to these functions: 

| self . set_session_variable( ) 

| self . get_session_variable( ) 

| self . ANY_METHOD_IN_COMMANDS_SUBCLASS -- e.g. execfunc from 

palantir /commands/Commands 

I 

| execute(self , opts, args) from tybase . palantir . commands . Commands 
| Runs a command (e.g. a new process) on the remote side. 
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| The elements of the command line are given as positional 

arguments in 

| *args. Specify your command-line already tokenized, as you 

would when 

| using python's subprocess module (which is what's used on the 

remote 

| side). 

I 

| opts and **kargs serve the same purpose. They let you specify 

keyword 

| arguments to affect how the command is run on the remote side. 

Valid 

| keywords and their meaning are as follows: 

| bool wait: whether to wait for the command to complete 

before 

| execute returns; defaults True 

| str cwd: current working directory for running the 

command; 

| defaults to the current working directory at the time 

of 

| invocation 

| str stdin: path to a file on the remote side from which to 

read 

| data to the process's standard in 

| str stdout: path to a file on the remote side to which to 

write 

| the process's standard out 

| str stderr: path to a file on the remote side to which to 

write 

| the process's standard error 
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| bool detach: whether to detach the remote process from the 

palantir 

| server which invoked it; defaults True 

| bool shell: if True, invoke the command in the shell; use 

this if 

| you need to use shell pipelining or for some other 

reason have 

| to specify the command as one big string; defaults 

False. If 

| setting shell True, you almost certainly want to 

specify your 

| command as one string rather than pre-tokenized . If 

you set 

| shell=True and specify a list of commands, behavior 

will differ 

| based on the target operating system; read the python 

subprocess 

| module documentation for more info. 

| The values in kargs are merged into opts, you can specify 

these options 

| as either elements of the opts diet, or as keyword arguments 

to execute. 


| fhash(self, fname, method= l md5' , offset=0, nbytes=0) from 
tybase . palantir . commands . Commands 

| Return hash via methods [md5|crc32] 


| fread(self, fname, offset=0, nbytes=0) from 
tybase . palantir . commands . Commands 

| Return file contents. 


| fwrite(self, fname, fdata, offset=0) from 
tybase . palantir . commands . Commands 
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| Write file contents. 

I 

| service_echo(self , *args, **kargs) from 
tybase . palantir . commands . Commands 

| Return arguments. 

I 

| service_log_level(self , log_level, *logs) from 
tybase . palantir . commands . Commands 

| Call setLogLevel on the root logger and any other named 

loggers 

I 

| service_ping(self ) from tybase . palantir . commands . Commands 
| Return True. 

I 

| service_secure_cert_fname(self ) from 
tybase . palantir . commands . Commands 

| Incoming format: 

I 

| Returns the string filename for the trusted SSL 

| certificate file. 

I 

| service_shutdown(self ) from tybase . palantir . commands . Commands 
| Shut the server down. 

I 

| service_timestamp(self ) from tybase . palantir . commands . Commands 
| Return service start time. 

I 

| service_uptime(self ) from tybase . palantir . commands . Commands 


UNCLASSIFIED 



UNCLASSIFIED 


Return service uptime. 

service_version(self ) from tybase . palantir . commands . Commands 
Return server version. 

shutdown(self ) from tybase . palantir . commands . Commands 
Shuts down the palantir server. 


Data descriptors defined here: 

diet 

dictionary for instance variables (if defined) 

weakref 

list of weak references to the object (if defined) 


7.1.8 plundermine 


Usage: plundermine [<option>]* <script_name>[, <script_name>] 
[host_slot]* [-- [arg_slot]*] 

host_slot = host[,host]* 

host = [ip | name | file : filename | - ] 

arg_slot = arg[,arg]* 
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Options : 


-h 


-v 


-V 

-q 

-1 <dir> 

-t <timeout> 
-p <port> 

-m <mode> 

-c <num> 


-d 

-S 

-N 


Print help. 

Enable verbose mode (script). 

Enable verbose mode (script + palantir). 
Disable verbose mode. 

Directory log files are located in. 
Timeout before exiting. 

Port to connect to. 

Recursively set mode on output directory. 
Maximum number of concurrent undermines. 
Debug mode (print undermines to execute). 
Use secure transport mode. 

Use non-secure transport mode. 


7.1.9 undermine 


Command-line interface to the undermine system. Allows running 
leafnodes 

with various settings, and also the interactive undermine shell. 

Usage: undermine [option]* <script_name> [host]* [-- [args]* 
[kwargs] *] 

Options : 

-h : Print help. 

-v : Enable verbose mode (script). 
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-V 

-q 

-1 <dir> 
output 


Enable verbose mode (script + palantir). 
Disable verbose mode. 

Set output directory (directory log files and 


files are written to), (string) 

-t <timeout> : Timeout before exiting, (int) 

-p <port> : Port to connect to. (int) 

-m <mode> : Recursively set mode on output directory, (string 

mode specification) 

-M : test and set dirty mark on hosts 

-s <sessionId> : Enable interactive shell and set first sessionld 

(if enabled, host, args, and kwargs are ignored) 

(string?) 

-X : Shut down the assets at the end of the tests 


args are specified in one of three forms, depending upon the 
characters 

preceding the value of the argument: 

Indicates a raw string, e.g: 

@@some_val 

@@'C:\Documents and SettingsXAll Users' 

Indicates the value is an expression which will be evaluated, 

e.g. : 

@True #(for a boolean) 

@100.5 #(for an float) 

@'"some string"' #(for a string, note how inner quoting is 

required 

since it will be evaluated) 
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@"['foo', True, 35]" #(for a list, explained further on) 

No preceding characters: Indicates a raw string, e.g.: 
some_val 

'some message string' 

kwargs are specified in the form name=value, where name is the name of 
the 

keyword argument being set, and value is the value specified just as 
with 

args above, e.g.: 
keep=@False 
delay=@25 

failure_msg= ' something went wrong' 

test_args=@" [ ' foo ' , 'bar', 'baz']" (for specifying a list) 
test_kwargs=@"{keyl=vall, key2=val2}" (for specifying a diet) 

Notes on quoting: 

There are two logical levels at which quotes may be required. 
First, you 

need quotes around values which have spaces in them to get those 
values 

into the argument parser intact. For example, the following two 
commands will see different arguments: 

bin/undermine some. script hostl -- 'foo bar' 
bin/undermine some. script hostl -- foo bar 
The first will see a single argument, 'foo bar'. The second will 

see 

two arguments, 'foo' and 'bar'. 
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Second, in evaluated arguments, you may need inner quotes so that 
the 

argument parser which evaluates your value knows how to behave. 

For 

example, if you want to specify a string as an evaluated argument, 

the 

command 

bin/undermine some. script hostl -- @'some string' 

will generate the error 

SyntaxError: Syntax error (line 1) 
p=LexToken( keyname, ' string ' , 1, 5) 

This is because the parser will effectively see that string as a 
line of 

code to be parsed, not the literal string "some string". Instead, 
the 

correct way to specify it is like so: 

bin/undermine some. script hostl -- @"'some string'" 

The outer set of double quotes gets the entire argument value into 
the 

parser intact, the inner quotes lets the parser know to treat it 

as a 

string literal. 


In list context, the parser has some intelligence and does not 

need 

string list items which have no spaces to be quoted. E.g., the 
following 

two commands will parse successfully and have the same effect: 
bin/undermine some. script hostl -- @['foo', 'bar', True, 1.5, 

'a b ' ] 

bin/undermine some. script hostl -- @[foo, bar, True, 1.5, 'a 
b'] 
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In both cases, undermine sees the arguments as a string literal 
'foo', a 

string literal 'bar', the python boolean True, the float 1.5, and 
a 

string literal ' a b ' . 

Raw strings and quotes: There is some unusual behavior with quotes 
in 

raw strings, where the parser removes quotes in some cases. I will 

illustrate this by examples. The commands 

bin/undermine some. script hostl -- @@['foo'] 
bin/undermine some. script hostl -- ['foo'] 

which you might expect to be able to do if you wanted the string 
literal 

"['foo']" will actually result in the argument 
[foo] 

The parser has stripped the outer set of matching quotes. 

Instead, to 

accomplish an argument of "['foo']", you would have to do one of 
the 

following commands 

bin/undermine some. script hostl -- @@"['foo']" 
bin/undermine some. script hostl -- @@["'foo'"] 
bin/undermine some. script hostl -- ["'foo'"] 
bin/undermine some. script hostl -- "['foo']" 
bin/undermine some. script hostl -- [VfooV] 

Notice how in the first four cases, the parser is stripping off 
the 

outer set of quotes. The fifth case illustrates that you can also 

escape the quotes to keep the parser from stripping them. 
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